Crate parquet_variant_compute 

VariantArray and compute kernels for the Variant Binary Encoding from Apache Parquet.

Main APIs

• VariantArray : Represents an array of Variant values.
• VariantArrayBuilder: For building VariantArray
• json_to_variant: Function to convert a batch of JSON strings to a VariantArray.
• variant_to_json: Function to convert a VariantArray to a batch of JSON strings.
• cast_to_variant: Module to cast other Arrow arrays to VariantArray.
• variant_get: Module to get values from a VariantArray using a specified VariantPath

🚧 Work In Progress

This crate is under active development and is not yet ready for production use. If you are interested in helping, you can find more information on the GitHub Variant issue

Re-exports

pub use cast_to_variant::cast_to_variant;

pub use cast_to_variant::cast_to_variant_with_options;

Modules

arrow_to_variant 🔒

cast_to_variant

from_json 🔒

Module for transforming a batch of JSON strings into a batch of Variants represented as STRUCT<metadata: BINARY, value: BINARY>

shred_variant 🔒

Module for shredding VariantArray with a given schema.

to_json 🔒

Module for transforming a batch of Variants represented as STRUCT<metadata: BINARY, value: BINARY> into a batch of JSON strings.

type_conversion 🔒

Module for transforming a typed arrow Array to VariantArray.

unshred_variant 🔒

Module for unshredding VariantArray by folding typed_value columns back into the value column.

variant_array 🔒

VariantArray implementation

variant_array_builder 🔒

VariantArrayBuilder implementation

variant_get

variant_to_arrow 🔒

Structs

BorrowedShreddingState

Similar to ShreddingState except it holds borrowed references of the target arrays. Useful for avoiding clone operations when the caller does not need a self-standing shredding state.

CastOptions

Options for controlling the behavior of cast_to_variant_with_options.

ShreddingState

Represents the shredding state of a VariantArray

VariantArray

An array of Parquet [Variant] values

VariantArrayBuilder

A builder for VariantArray

VariantType

Arrow Variant [ExtensionType].

VariantValueArrayBuilder

A builder for creating only the value column of a VariantArray

Functions

json_to_variant

Parse a batch of JSON strings into a batch of Variants represented as STRUCT<metadata: BINARY, value: BINARY> where nulls are preserved. The JSON strings in the input must be valid.

shred_variant

Shreds the input binary variant using a target shredding schema derived from the requested data type.

unshred_variant

Removes all (nested) typed_value columns from a VariantArray by converting them back to binary variant and merging the resulting values back into the value column.

variant_to_json

Transform a batch of Variant represented as STRUCT<metadata: BINARY, value: BINARY> to a batch of JSON strings where nulls are preserved. The JSON strings in the input must be valid.