org.apache.spark.sql

Class CarbonVectorProxy

java.lang.Object

org.apache.spark.sql.CarbonVectorProxy

public class CarbonVectorProxy
extends Object

Adapter class which handles the columnar vector reading of the carbondata based on the spark ColumnVector and ColumnarBatch API. This proxy class handles the complexity of spark 2.3 version related api changes since spark ColumnVector and ColumnarBatch interfaces are still evolving.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	CarbonVectorProxy.ColumnVectorProxy
static class	CarbonVectorProxy.ColumnVectorProxyWithLazyLoad

Constructor Summary

Constructors

Constructor and Description
CarbonVectorProxy(org.apache.spark.memory.MemoryMode memMode, org.apache.spark.sql.types.StructType outputSchema, int rowNum, boolean useLazyLoad) Adapter class which handles the columnar vector reading of the carbondata based on the spark ColumnVector and ColumnarBatch API.

Method Summary

Modifier and Type	Method and Description
void	close() Called to close all the columns in this batch.
org.apache.spark.sql.execution.vectorized.ColumnVector	column(int ordinal) This API will return a columnvector from a batch of column vector rows based on the ordinal
org.apache.spark.sql.types.DataType	dataType(int ordinal)
Object	getColumnarBatch() Returns the row in this batch at `rowId`.
CarbonVectorProxy.ColumnVectorProxy	getColumnVector(int ordinal)
org.apache.spark.sql.catalyst.InternalRow	getRow(int rowId) Returns the row in this batch at `rowId`.
int	numRows() Returns the number of rows for read, including filtered rows.
void	reset() Resets this column for writing.
void	resetDictionaryIds(int ordinal)
void	setNumRows(int numRows) Sets the number of rows in this batch.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

CarbonVectorProxy

public CarbonVectorProxy(org.apache.spark.memory.MemoryMode memMode,
                         org.apache.spark.sql.types.StructType outputSchema,
                         int rowNum,
                         boolean useLazyLoad)

Adapter class which handles the columnar vector reading of the carbondata based on the spark ColumnVector and ColumnarBatch API. This proxy class handles the complexity of spark 2.3 version related api changes since spark ColumnVector and ColumnarBatch interfaces are still evolving.

Parameters:

memMode - which represent the type onheap or offheap vector.

outputSchema, - metadata related to current schema of table.

rowNum - rows number for vector reading

useLazyLoad - Whether to use lazy load while getting the data.

Method Detail

getColumnVector

public CarbonVectorProxy.ColumnVectorProxy getColumnVector(int ordinal)

numRows

public int numRows()

Returns the number of rows for read, including filtered rows.

column

public org.apache.spark.sql.execution.vectorized.ColumnVector column(int ordinal)

This API will return a columnvector from a batch of column vector rows based on the ordinal

Parameters:

ordinal -

Returns:

reset

public void reset()

Resets this column for writing. The currently stored values are no longer accessible.

resetDictionaryIds

public void resetDictionaryIds(int ordinal)

getRow

public org.apache.spark.sql.catalyst.InternalRow getRow(int rowId)

Returns the row in this batch at `rowId`. Returned row is reused across calls.

getColumnarBatch

public Object getColumnarBatch()

Returns the row in this batch at `rowId`. Returned row is reused across calls.

close

public void close()

Called to close all the columns in this batch. It is not valid to access the data after calling this. This must be called at the end to clean up memory allocations.

setNumRows

public void setNumRows(int numRows)

Sets the number of rows in this batch.

dataType

public org.apache.spark.sql.types.DataType dataType(int ordinal)