Package org.apache.beam.sdk.extensions.arrow

Class ArrowConversion

java.lang.Object

org.apache.beam.sdk.extensions.arrow.ArrowConversion

public class ArrowConversion
extends java.lang.Object

Utilities to create Iterables of Beam Row instances backed by Arrow record batches.

Nested Class Summary

Nested Classes

Modifier and Type	Class	Description
static class	ArrowConversion.ArrowSchemaTranslator	Converts Arrow schema to Beam row schema.
static class	ArrowConversion.RecordBatchRowIterator

Method Summary

Modifier and Type	Method	Description
static org.apache.arrow.vector.types.pojo.Schema	arrowSchemaFromInput(java.io.InputStream input)
static ArrowConversion.RecordBatchRowIterator	rowsFromRecordBatch(org.apache.beam.sdk.schemas.Schema schema, org.apache.arrow.vector.VectorSchemaRoot vectorSchemaRoot)	Returns a ArrowConversion.RecordBatchRowIterator backed by the Arrow record batch stored in vectorSchemaRoot.
static ArrowConversion.RecordBatchRowIterator	rowsFromSerializedRecordBatch(org.apache.arrow.vector.types.pojo.Schema arrowSchema, java.io.InputStream inputStream, org.apache.arrow.memory.RootAllocator allocator)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Method Detail

rowsFromRecordBatch

public static ArrowConversion.RecordBatchRowIterator rowsFromRecordBatch(org.apache.beam.sdk.schemas.Schema schema,
                                                                         org.apache.arrow.vector.VectorSchemaRoot vectorSchemaRoot)

Returns a ArrowConversion.RecordBatchRowIterator backed by the Arrow record batch stored in vectorSchemaRoot.

Note this is a lazy interface. The data in the underlying Arrow buffer is not read until a field of one of the returned Rows is accessed.

rowsFromSerializedRecordBatch

public static ArrowConversion.RecordBatchRowIterator rowsFromSerializedRecordBatch(org.apache.arrow.vector.types.pojo.Schema arrowSchema,
                                                                                   java.io.InputStream inputStream,
                                                                                   org.apache.arrow.memory.RootAllocator allocator)
                                                                            throws java.io.IOException

Throws:

java.io.IOException

arrowSchemaFromInput

public static org.apache.arrow.vector.types.pojo.Schema arrowSchemaFromInput(java.io.InputStream input)
                                                                      throws java.io.IOException

Throws:

java.io.IOException