Crate parquet_variant_compute 

VariantArray and compute kernels for the Variant Binary Encoding from Apache Parquet.

Main APIs

• VariantArray : Represents an array of Variant values.
• VariantArrayBuilder: For building VariantArray

Compute Kernels

• json_to_variant(): Function to convert Arrays of JSON strings to a VariantArray.
• variant_to_json(): Function to convert a VariantArray to arrays of JSON strings.
• cast_to_variant(): Cast Arrow arrays to VariantArray.
• variant_get(): Convert VariantArray (or an inner path) to a strongly-typed Arrow array.
• shred_variant(): Shred a VariantArray according to the provided shredding schema
• unshred_variant(): Unshred a VariantArray to pure binary variant.

🚧 Work In Progress

This crate is under active development and is not yet ready for production use. If you are interested in helping, you can find more information on the GitHub Variant issue

Platform Support

Only little-endian platforms are officially supported and tested in CI. Big-endian platforms are not tested in CI and may not work correctly. Fixes for big-endian platforms are welcome and handled on a best-effort basis, but compatibility is not guaranteed.

Re-exports

pub use variant_array::ShreddingState;

pub use variant_array::VariantArray;

pub use variant_array::VariantArrayIter;

pub use variant_array_builder::VariantArrayBuilder;

pub use variant_array_builder::VariantValueArrayBuilder;

pub use cast_to_variant::cast_to_variant;

pub use cast_to_variant::cast_to_variant_with_options;

pub use from_json::json_to_variant;

pub use shred_variant::IntoShreddingField;

pub use shred_variant::ShreddedSchemaBuilder;

pub use shred_variant::shred_variant;

pub use to_json::variant_to_json;

pub use unshred_variant::unshred_variant;

pub use variant_get::GetOptions;

pub use variant_get::variant_get;

Modules

arrow_to_variant 🔒

cast_to_variant 🔒

from_json 🔒

Module for transforming a batch of JSON strings into a batch of Variants represented as STRUCT<metadata: BINARY, value: BINARY>

shred_variant 🔒

Module for shredding VariantArray with a given schema.

to_json 🔒

Module for transforming a batch of Variants into JSON strings.

type_conversion 🔒

Module for transforming a typed arrow Array to VariantArray.

unshred_variant 🔒

Module for unshredding VariantArray by folding typed_value columns back into the value column.

variant_array 🔒

VariantArray implementation

variant_array_builder 🔒

VariantArrayBuilder implementation

variant_get 🔒

variant_to_arrow 🔒

Structs

VariantType

Arrow Variant [ExtensionType].