Class Data

java.lang.Object
org.apache.arrow.c.Data
public final class Data extends Object
Functions for working with the C data interface.

This API is EXPERIMENTAL. Note that currently only 64bit systems are supported.

Importing ArrowArrays with a non-zero offset is not always zero-copy: buffers that cannot be sliced at the offset, such as offset buffers or view data, are copied. Importing such an array into an existing vector fails if an extension vector nested in a list, map, union or other vector that is not a struct does not report its extension type in ValueVector.getField().

  • Method Details

    • exportField

      public static void exportField(BufferAllocator allocator, Field field, DictionaryProvider provider, ArrowSchema out)
      Export Java Field using the C data interface format.
      Parameters:
      allocator - Buffer allocator for allocating C data interface fields
      field - Field object to export
      provider - Dictionary provider for dictionary encoded fields (optional)
      out - C struct where to export the field

    • exportSchema

      public static void exportSchema(BufferAllocator allocator, Schema schema, DictionaryProvider provider, ArrowSchema out)
      Export Java Schema using the C data interface format.
      Parameters:
      allocator - Buffer allocator for allocating C data interface fields
      schema - Schema object to export
      provider - Dictionary provider for dictionary encoded fields (optional)
      out - C struct where to export the field

    • exportVector

      public static void exportVector(BufferAllocator allocator, FieldVector vector, DictionaryProvider provider, ArrowArray out)
      Export Java FieldVector using the C data interface format.

      The resulting ArrowArray struct keeps the array data and buffers alive until its release callback is called by the consumer.

      Parameters:
      allocator - Buffer allocator for allocating C data interface fields
      vector - Vector object to export
      provider - Dictionary provider for dictionary encoded vectors (optional)
      out - C struct where to export the array

    • exportVector

      public static void exportVector(BufferAllocator allocator, FieldVector vector, DictionaryProvider provider, ArrowArray out, ArrowSchema outSchema)
      Export Java FieldVector using the C data interface format.

      The resulting ArrowArray struct keeps the array data and buffers alive until its release callback is called by the consumer.

      Parameters:
      allocator - Buffer allocator for allocating C data interface fields
      vector - Vector object to export
      provider - Dictionary provider for dictionary encoded vectors (optional)
      out - C struct where to export the array
      outSchema - C struct where to export the array type (optional)

    • exportTable

      public static void exportTable(BufferAllocator allocator, Table table, ArrowArray out)
      Export the current contents of a Java Table using the C data interface format.

      The table is exported as if it were a struct array. The resulting ArrowArray struct keeps the record batch data and buffers alive until its release callback is called by the consumer.

      Parameters:
      allocator - Buffer allocator for allocating C data interface fields
      table - Table to export
      out - C struct where to export the record batch

    • exportTable

      public static void exportTable(BufferAllocator allocator, Table table, DictionaryProvider provider, ArrowArray out)
      Export the current contents of a Java Table using the C data interface format.

      The table is exported as if it were a struct array. The resulting ArrowArray struct keeps the record batch data and buffers alive until its release callback is called by the consumer.

      Parameters:
      allocator - Buffer allocator for allocating C data interface fields
      table - Table to export
      provider - Dictionary provider for dictionary encoded vectors (optional)
      out - C struct where to export the record batch

    • exportTable

      public static void exportTable(BufferAllocator allocator, Table table, DictionaryProvider provider, ArrowArray out, ArrowSchema outSchema)
      Export the current contents of a Java Table using the C data interface format.

      The table is exported as if it were a struct array. The resulting ArrowArray struct keeps the record batch data and buffers alive until its release callback is called by the consumer.

      Parameters:
      allocator - Buffer allocator for allocating C data interface fields
      table - Table to export
      provider - Dictionary provider for dictionary encoded vectors (optional)
      out - C struct where to export the record batch
      outSchema - C struct where to export the record batch schema (optional)

    • exportVectorSchemaRoot

      public static void exportVectorSchemaRoot(BufferAllocator allocator, VectorSchemaRoot vsr, DictionaryProvider provider, ArrowArray out)
      Export the current contents of a Java VectorSchemaRoot using the C data interface format.

      The vector schema root is exported as if it were a struct array. The resulting ArrowArray struct keeps the record batch data and buffers alive until its release callback is called by the consumer.

      Parameters:
      allocator - Buffer allocator for allocating C data interface fields
      vsr - Vector schema root to export
      provider - Dictionary provider for dictionary encoded vectors (optional)
      out - C struct where to export the record batch

    • exportVectorSchemaRoot

      public static void exportVectorSchemaRoot(BufferAllocator allocator, VectorSchemaRoot vsr, DictionaryProvider provider, ArrowArray out, ArrowSchema outSchema)
      Export the current contents of a Java VectorSchemaRoot using the C data interface format.

      The vector schema root is exported as if it were a struct array. The resulting ArrowArray struct keeps the record batch data and buffers alive until its release callback is called by the consumer.

      Parameters:
      allocator - Buffer allocator for allocating C data interface fields
      vsr - Vector schema root to export
      provider - Dictionary provider for dictionary encoded vectors (optional)
      out - C struct where to export the record batch
      outSchema - C struct where to export the record batch schema (optional)

    • exportArrayStream

      public static void exportArrayStream(BufferAllocator allocator, ArrowReader reader, ArrowArrayStream out)
      Export a reader as an ArrowArrayStream using the C Stream Interface.
      Parameters:
      allocator - Buffer allocator for allocating C data interface fields
      reader - Reader to export
      out - C struct to export the stream

    • importField

      public static Field importField(BufferAllocator allocator, ArrowSchema schema, CDataDictionaryProvider provider)
      Equivalent to calling importField(allocator, schema, provider, true).
      Parameters:
      allocator - Buffer allocator for allocating dictionary vectors
      schema - C data interface struct representing the field [inout]
      provider - A dictionary provider will be initialized with empty dictionary vectors (optional)
      Returns:
      Imported field object
      See Also:

    • importField

      public static Field importField(BufferAllocator allocator, ArrowSchema schema, CDataDictionaryProvider provider, boolean closeImportedStructs)
      Import Java Field from the C data interface.

      The given ArrowSchema struct is released (as per the C data interface specification), even if this function fails.

      Parameters:
      allocator - Buffer allocator for allocating dictionary vectors
      schema - C data interface struct representing the field [inout]
      provider - A dictionary provider will be initialized with empty dictionary vectors (optional)
      closeImportedStructs - if true, the ArrowSchema struct will be closed when this method completes.
      Returns:
      Imported field object

    • importSchema

      public static Schema importSchema(BufferAllocator allocator, ArrowSchema schema, CDataDictionaryProvider provider)
      Equivalent to calling importSchema(allocator, schema, provider, true).
      Parameters:
      allocator - Buffer allocator for allocating dictionary vectors
      schema - C data interface struct representing the field
      provider - A dictionary provider will be initialized with empty dictionary vectors (optional)
      Returns:
      Imported schema object
      See Also:

    • importSchema

      public static Schema importSchema(BufferAllocator allocator, ArrowSchema schema, CDataDictionaryProvider provider, boolean closeImportedStructs)
      Import Java Schema from the C data interface.

      The given ArrowSchema struct is released (as per the C data interface specification), even if this function fails.

      Parameters:
      allocator - Buffer allocator for allocating dictionary vectors
      schema - C data interface struct representing the field
      provider - A dictionary provider will be initialized with empty dictionary vectors (optional)
      closeImportedStructs - if true, the ArrowSchema struct will be closed when this method completes.
      Returns:
      Imported schema object

    • importIntoVector

      public static void importIntoVector(BufferAllocator allocator, ArrowArray array, FieldVector vector, DictionaryProvider provider)
      Equivalent to calling importIntoVector(BufferAllocator, ArrowArray, FieldVector, DictionaryProvider, boolean) importIntoVector(allocator, array, vector, provider, true)}.
      Parameters:
      allocator - Buffer allocator
      array - C data interface struct holding the array data
      vector - Imported vector object [out]
      provider - Dictionary provider to load dictionary vectors to (optional)
      See Also:

    • importIntoVector

      public static void importIntoVector(BufferAllocator allocator, ArrowArray array, FieldVector vector, DictionaryProvider provider, boolean closeImportedStructs)
      Import Java vector from the C data interface.

      On successful completion, the ArrowArray struct will have been moved (as per the C data interface specification) to a private object held alive by the resulting array.

      Parameters:
      allocator - Buffer allocator
      array - C data interface struct holding the array data
      vector - Imported vector object [out]
      provider - Dictionary provider to load dictionary vectors to (optional)
      closeImportedStructs - if true, the ArrowArray struct will be closed when this method completes successfully.

    • importVector

      public static FieldVector importVector(BufferAllocator allocator, ArrowArray array, ArrowSchema schema, CDataDictionaryProvider provider)
      Equivalent to calling importVector(allocator, array, schema, provider, true).
      Parameters:
      allocator - Buffer allocator for allocating the output FieldVector
      array - C data interface struct holding the array data
      schema - C data interface struct holding the array type
      provider - Dictionary provider to load dictionary vectors to (optional)
      Returns:
      Imported vector object
      See Also:

    • importVector

      public static FieldVector importVector(BufferAllocator allocator, ArrowArray array, ArrowSchema schema, CDataDictionaryProvider provider, boolean closeImportedStructs)
      Import Java vector and its type from the C data interface.

      The ArrowArray struct has its contents moved (as per the C data interface specification) to a private object held alive by the resulting vector. The ArrowSchema struct is released, even if this function fails.

      Parameters:
      allocator - Buffer allocator for allocating the output FieldVector
      array - C data interface struct holding the array data
      schema - C data interface struct holding the array type
      provider - Dictionary provider to load dictionary vectors to (optional)
      closeImportedStructs - if true, the ArrowArray struct will be closed when this method completes successfully and the ArrowSchema struct will be always be closed.
      Returns:
      Imported vector object

    • importIntoVectorSchemaRoot

      public static void importIntoVectorSchemaRoot(BufferAllocator allocator, ArrowArray array, VectorSchemaRoot root, DictionaryProvider provider)
      Equivalent to calling importIntoVectorSchemaRoot(allocator, array, root, provider, true).
      Parameters:
      allocator - Buffer allocator
      array - C data interface struct holding the record batch data
      root - vector schema root to load into
      provider - Dictionary provider to load dictionary vectors to (optional)
      See Also:

    • importIntoVectorSchemaRoot

      public static void importIntoVectorSchemaRoot(BufferAllocator allocator, ArrowArray array, VectorSchemaRoot root, DictionaryProvider provider, boolean closeImportedStructs)
      Import record batch from the C data interface into vector schema root.

      The ArrowArray struct has its contents moved (as per the C data interface specification) to a private object held alive by the resulting vector schema root.

      The schema of the vector schema root must match the input array (undefined behavior otherwise).

      Parameters:
      allocator - Buffer allocator
      array - C data interface struct holding the record batch data
      root - vector schema root to load into
      provider - Dictionary provider to load dictionary vectors to (optional)
      closeImportedStructs - if true, the ArrowArray struct will be closed when this method completes successfully

    • importVectorSchemaRoot

      public static VectorSchemaRoot importVectorSchemaRoot(BufferAllocator allocator, ArrowSchema schema, CDataDictionaryProvider provider)
      Equivalent to calling importVectorSchemaRoot(allocator, schema, provider, true).
      Parameters:
      allocator - Buffer allocator for allocating the output VectorSchemaRoot
      schema - C data interface struct holding the record batch schema
      provider - Dictionary provider to load dictionary vectors to (optional)
      Returns:
      Imported vector schema root
      See Also:

    • importVectorSchemaRoot

      public static VectorSchemaRoot importVectorSchemaRoot(BufferAllocator allocator, ArrowSchema schema, CDataDictionaryProvider provider, boolean closeImportedStructs)
      Import Java vector schema root from a C data interface Schema.

      The type represented by the ArrowSchema struct must be a struct type array.

      The ArrowSchema struct is released, even if this function fails.

      Parameters:
      allocator - Buffer allocator for allocating the output VectorSchemaRoot
      schema - C data interface struct holding the record batch schema
      provider - Dictionary provider to load dictionary vectors to (optional)
      closeImportedStructs - if true, the ArrowSchema struct will be closed when this method completes
      Returns:
      Imported vector schema root

    • importVectorSchemaRoot

      public static VectorSchemaRoot importVectorSchemaRoot(BufferAllocator allocator, ArrowArray array, ArrowSchema schema, CDataDictionaryProvider provider)
      Equivalent to calling importVectorSchemaRoot(allocator, array, schema, provider, true).
      Parameters:
      allocator - Buffer allocator for allocating the output VectorSchemaRoot
      array - C data interface struct holding the record batch data (optional)
      schema - C data interface struct holding the record batch schema
      provider - Dictionary provider to load dictionary vectors to (optional)
      Returns:
      Imported vector schema root
      See Also:

    • importVectorSchemaRoot

      public static VectorSchemaRoot importVectorSchemaRoot(BufferAllocator allocator, ArrowArray array, ArrowSchema schema, CDataDictionaryProvider provider, boolean closeImportedStructs)
      Import Java vector schema root from the C data interface.

      The type represented by the ArrowSchema struct must be a struct type array.

      The ArrowArray struct has its contents moved (as per the C data interface specification) to a private object held alive by the resulting record batch. The ArrowSchema struct is released, even if this function fails.

      Prefer importIntoVectorSchemaRoot(BufferAllocator, ArrowArray, VectorSchemaRoot, DictionaryProvider) for loading array data while reusing the same vector schema root.

      Parameters:
      allocator - Buffer allocator for allocating the output VectorSchemaRoot
      array - C data interface struct holding the record batch data (optional)
      schema - C data interface struct holding the record batch schema
      provider - Dictionary provider to load dictionary vectors to (optional)
      closeImportedStructs - if true, the ArrowArray struct will be closed when this method completes successfully and the ArrowSchema struct will be always be closed.
      Returns:
      Imported vector schema root

    • importArrayStream

      public static ArrowReader importArrayStream(BufferAllocator allocator, ArrowArrayStream stream)
      Equivalent to calling importArrayStream(allocator, stream, true).
      Parameters:
      allocator - Buffer allocator for allocating the output data.
      stream - C stream interface struct to import.
      Returns:
      Imported reader
      See Also:

    • importArrayStream

      public static ArrowReader importArrayStream(BufferAllocator allocator, ArrowArrayStream stream, boolean closeImportedStructs)
      Import an ArrowArrayStream as an ArrowReader.

      On successful completion, the ArrowArrayStream struct will have been moved (as per the C data interface specification) to a private object held alive by the resulting ArrowReader.

      Parameters:
      allocator - Buffer allocator for allocating the output data.
      stream - C stream interface struct to import.
      closeImportedStructs - if true, the ArrowArrayStream struct will be closed when this method completes successfully
      Returns:
      Imported reader