Crate parquet_variant_compute 

VariantArray and compute kernels for the Variant Binary Encoding from Apache Parquet.

Main APIs

• VariantArray : Represents an array of Variant values.
• VariantArrayBuilder: For building VariantArray

Compute Kernels

• json_to_variant(): Function to convert Arrays of JSON strings to a VariantArray.
• variant_to_json(): Function to convert a VariantArray to arrays of JSON strings.
• cast_to_variant(): Cast Arrow arrays to VariantArray.
• variant_get(): Convert VariantArray (or an inner path) to a strongly-typed Arrow array.
• shred_variant(): Shred a VariantArray according to the provided shredding schema
• unshred_variant(): Unshred a VariantArray to pure binary variant.

🚧 Work In Progress

This crate is under active development and is not yet ready for production use. If you are interested in helping, you can find more information on the GitHub Variant issue

Modules

arrow_to_variant 🔒

cast_to_variant 🔒

from_json 🔒

Module for transforming a batch of JSON strings into a batch of Variants represented as STRUCT<metadata: BINARY, value: BINARY>

shred_variant 🔒

Module for shredding VariantArray with a given schema.

to_json 🔒

Module for transforming a batch of Variants represented as STRUCT<metadata: BINARY, value: BINARY> into a batch of JSON strings.

type_conversion 🔒

Module for transforming a typed arrow Array to VariantArray.

unshred_variant 🔒

Module for unshredding VariantArray by folding typed_value columns back into the value column.

variant_array 🔒

VariantArray implementation

variant_array_builder 🔒

VariantArrayBuilder implementation

variant_get 🔒

variant_to_arrow 🔒

Structs

BorrowedShreddingState

Similar to ShreddingState except it holds borrowed references of the target arrays. Useful for avoiding clone operations when the caller does not need a self-standing shredding state.

CastOptions

Options for controlling the behavior of cast_to_variant_with_options.

GetOptions

Controls the action of the variant_get kernel.

ShreddingState

Represents the shredding state of a VariantArray

VariantArray

An array of Parquet [Variant] values

VariantArrayBuilder

A builder for VariantArray

VariantType

Arrow Variant [ExtensionType].

VariantValueArrayBuilder

A builder for creating only the value column of a VariantArray

Functions

cast_to_variant

Convert an array to a VariantArray with strict mode enabled (returns errors on conversion failures).

cast_to_variant_with_options

Casts a typed arrow [Array] to a VariantArray. This is useful when you need to convert a specific data type

json_to_variant

Parse a batch of JSON strings into a batch of Variants represented as STRUCT<metadata: BINARY, value: BINARY> where nulls are preserved. The JSON strings in the input must be valid.

shred_variant

Shreds the input binary variant using a target shredding schema derived from the requested data type.

unshred_variant

Removes all (nested) typed_value columns from a VariantArray by converting them back to binary variant and merging the resulting values back into the value column.

variant_get

Returns an array with the specified path extracted from the variant values.

variant_to_json

Transform a batch of Variant represented as STRUCT<metadata: BINARY, value: BINARY> to a batch of JSON strings where nulls are preserved. The JSON strings in the input must be valid.