parquet.hadoop

Class ParquetInputSplit

java.lang.Object

org.apache.hadoop.mapreduce.InputSplit

parquet.hadoop.ParquetInputSplit

All Implemented Interfaces:

org.apache.hadoop.io.Writable

public class ParquetInputSplit
extends org.apache.hadoop.mapreduce.InputSplit
implements org.apache.hadoop.io.Writable

An input split for the Parquet format It contains the information to read one block of the file.

Author:

Julien Le Dem

Constructor Summary

Constructors

Constructor and Description
ParquetInputSplit() Writables must have a parameterless constructor
ParquetInputSplit(org.apache.hadoop.fs.Path path, long start, long length, String[] hosts, List<BlockMetaData> blocks, String requestedSchema, String fileSchema, Map<String,String> extraMetadata, Map<String,String> readSupportMetadata) Used by ParquetInputFormat.getSplits(org.apache.hadoop.mapreduce.JobContext)

Method Summary

Methods

Modifier and Type	Method and Description
List<BlockMetaData>	getBlocks()
Map<String,String>	getExtraMetadata()
String	getFileSchema()
long	getLength()
String[]	getLocations()
org.apache.hadoop.fs.Path	getPath()
Map<String,String>	getReadSupportMetadata()
String	getRequestedSchema()
long	getStart()
void	readFields(DataInput in)
String	toString()
void	write(DataOutput out)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Constructor Detail

ParquetInputSplit

public ParquetInputSplit()

Writables must have a parameterless constructor

ParquetInputSplit

public ParquetInputSplit(org.apache.hadoop.fs.Path path,
                 long start,
                 long length,
                 String[] hosts,
                 List<BlockMetaData> blocks,
                 String requestedSchema,
                 String fileSchema,
                 Map<String,String> extraMetadata,
                 Map<String,String> readSupportMetadata)

Used by ParquetInputFormat.getSplits(org.apache.hadoop.mapreduce.JobContext)

Parameters:

path - the path to the file

start - the offset of the block in the file

length - the size of the block in the file

hosts - the hosts where this block can be found

blocks - the block meta data (Columns locations)

schema - the file schema

readSupportClass - the class used to materialize records

requestedSchema - the requested schema for materialization

fileSchema - the schema of the file

extraMetadata - the app specific meta data in the file

readSupportMetadata - the read support specific metadata

Method Detail

getBlocks

public List<BlockMetaData> getBlocks()

Returns:

the block meta data

getLength

public long getLength()
               throws IOException,
                      InterruptedException

Specified by:

getLength in class org.apache.hadoop.mapreduce.InputSplit

Throws:

IOException

InterruptedException

getLocations

public String[] getLocations()
                      throws IOException,
                             InterruptedException

Specified by:

getLocations in class org.apache.hadoop.mapreduce.InputSplit

Throws:

IOException

InterruptedException

getStart

public long getStart()

Returns:

the offset of the block in the file

getPath

public org.apache.hadoop.fs.Path getPath()

Returns:

the path of the file containing the block

getRequestedSchema

public String getRequestedSchema()

Returns:

the requested schema

getFileSchema

public String getFileSchema()

Returns:

the file schema

getExtraMetadata

public Map<String,String> getExtraMetadata()

Returns:

app specific metadata from the file

getReadSupportMetadata

public Map<String,String> getReadSupportMetadata()

Returns:

app specific metadata provided by the read support in the init phase

readFields

public void readFields(DataInput in)
                throws IOException

Specified by:

readFields in interface org.apache.hadoop.io.Writable

Throws:

IOException

write

public void write(DataOutput out)
           throws IOException

Specified by:

write in interface org.apache.hadoop.io.Writable

Throws:

IOException

toString

public String toString()

Overrides:

toString in class Object