parquet.hadoop

Class ParquetWriter<T>

java.lang.Object

parquet.hadoop.ParquetWriter<T>

All Implemented Interfaces:

Closeable, AutoCloseable

public class ParquetWriter<T>
extends Object
implements Closeable

Write records to a Parquet file.

Field Summary

Fields

Modifier and Type	Field and Description
static int	DEFAULT_BLOCK_SIZE
static int	DEFAULT_PAGE_SIZE

Constructor Summary

Constructors

Constructor and Description
ParquetWriter(org.apache.hadoop.fs.Path file, WriteSupport<T> writeSupport) Create a new ParquetWriter.
ParquetWriter(org.apache.hadoop.fs.Path file, WriteSupport<T> writeSupport, CompressionCodecName compressionCodecName, int blockSize, int pageSize) Create a new ParquetWriter.
ParquetWriter(org.apache.hadoop.fs.Path file, WriteSupport<T> writeSupport, CompressionCodecName compressionCodecName, int blockSize, int pageSize, boolean enableDictionary, boolean validating) Create a new ParquetWriter.
ParquetWriter(org.apache.hadoop.fs.Path file, WriteSupport<T> writeSupport, CompressionCodecName compressionCodecName, int blockSize, int pageSize, int dictionaryPageSize, boolean enableDictionary, boolean validating) Create a new ParquetWriter.

Method Summary

Methods

Modifier and Type	Method and Description
void	close()
void	write(T object)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

DEFAULT_BLOCK_SIZE

public static final int DEFAULT_BLOCK_SIZE

See Also:

Constant Field Values

DEFAULT_PAGE_SIZE

public static final int DEFAULT_PAGE_SIZE

See Also:

Constant Field Values

Constructor Detail

ParquetWriter

public ParquetWriter(org.apache.hadoop.fs.Path file,
             WriteSupport<T> writeSupport,
             CompressionCodecName compressionCodecName,
             int blockSize,
             int pageSize)
              throws IOException

Create a new ParquetWriter. (with dictionary encoding enabled and validation off)

Parameters:

file - the file to create

writeSupport - the implementation to write a record to a RecordConsumer

compressionCodecName - the compression codec to use

blockSize - the block size threshold

pageSize - the page size threshold

Throws:

IOException

See Also:

ParquetWriter#ParquetWriter(Path, WriteSupport, CompressionCodecName, int, int, boolean)

ParquetWriter

public ParquetWriter(org.apache.hadoop.fs.Path file,
             WriteSupport<T> writeSupport,
             CompressionCodecName compressionCodecName,
             int blockSize,
             int pageSize,
             boolean enableDictionary,
             boolean validating)
              throws IOException

Create a new ParquetWriter.

Parameters:

file - the file to create

writeSupport - the implementation to write a record to a RecordConsumer

compressionCodecName - the compression codec to use

blockSize - the block size threshold

pageSize - the page size threshold (both data and dictionary)

enableDictionary - to turn dictionary encoding on

validating - to turn on validation using the schema

Throws:

IOException

ParquetWriter

public ParquetWriter(org.apache.hadoop.fs.Path file,
             WriteSupport<T> writeSupport,
             CompressionCodecName compressionCodecName,
             int blockSize,
             int pageSize,
             int dictionaryPageSize,
             boolean enableDictionary,
             boolean validating)
              throws IOException

Create a new ParquetWriter.

Parameters:

file - the file to create

writeSupport - the implementation to write a record to a RecordConsumer

compressionCodecName - the compression codec to use

blockSize - the block size threshold

pageSize - the page size threshold

dictionaryPageSize - the page size threshold for the dictionary pages

enableDictionary - to turn dictionary encoding on

validating - to turn on validation using the schema

Throws:

IOException

ParquetWriter

public ParquetWriter(org.apache.hadoop.fs.Path file,
             WriteSupport<T> writeSupport)
              throws IOException

Create a new ParquetWriter. The default block size is 50 MB.The default page size is 1 MB. Default compression is no compression. Dictionary encoding is disabled.

Parameters:

file - the file to create

writeSupport - the implementation to write a record to a RecordConsumer

Throws:

IOException

Method Detail

write

public void write(T object)
           throws IOException

Throws:

IOException

close

public void close()
           throws IOException

Specified by:

close in interface Closeable

Specified by:

close in interface AutoCloseable

Throws:

IOException