Expand description
VariantArray and compute kernels for the Variant Binary Encoding from Apache Parquet.
ยงMain APIs
VariantArray: Represents an array ofVariantvalues.VariantArrayBuilder: For buildingVariantArray
ยงCompute Kernels
json_to_variant(): Function to convert Arrays of JSON strings to aVariantArray.variant_to_json(): Function to convert aVariantArrayto arrays of JSON strings.cast_to_variant(): Cast Arrow arrays toVariantArray.variant_get(): ConvertVariantArray(or an inner path) to a strongly-typed Arrow array.shred_variant(): Shred aVariantArrayaccording to the provided shredding schemaunshred_variant(): Unshred aVariantArrayto pure binary variant.
ยง๐ง Work In Progress
This crate is under active development and is not yet ready for production use. If you are interested in helping, you can find more information on the GitHub Variant issue
Modulesยง
- arrow_
to_ ๐variant - cast_
to_ ๐variant - from_
json ๐ - Module for transforming a batch of JSON strings into a batch of Variants represented as STRUCT<metadata: BINARY, value: BINARY>
- shred_
variant ๐ - Module for shredding VariantArray with a given schema.
- to_json ๐
- Module for transforming a batch of Variants represented as STRUCT<metadata: BINARY, value: BINARY> into a batch of JSON strings.
- type_
conversion ๐ - Module for transforming a typed arrow
ArraytoVariantArray. - unshred_
variant ๐ - Module for unshredding VariantArray by folding typed_value columns back into the value column.
- variant_
array ๐ VariantArrayimplementation- variant_
array_ ๐builder VariantArrayBuilderimplementation- variant_
get ๐ - variant_
to_ ๐arrow
Structsยง
- Borrowed
Shredding State - Similar to
ShreddingStateexcept it holds borrowed references of the target arrays. Useful for avoiding clone operations when the caller does not need a self-standing shredding state. - Cast
Options - Options for controlling the behavior of
cast_to_variant_with_options. - GetOptions
- Controls the action of the variant_get kernel.
- Shredding
State - Represents the shredding state of a
VariantArray - Variant
Array - An array of Parquet [
Variant] values - Variant
Array Builder - A builder for
VariantArray - Variant
Type - Arrow Variant [
ExtensionType]. - Variant
Value Array Builder - A builder for creating only the value column of a
VariantArray
Functionsยง
- cast_
to_ variant - Convert an array to a
VariantArraywith strict mode enabled (returns errors on conversion failures). - cast_
to_ variant_ with_ options - Casts a typed arrow [
Array] to aVariantArray. This is useful when you need to convert a specific data type - json_
to_ variant - Parse a batch of JSON strings into a batch of Variants represented as STRUCT<metadata: BINARY, value: BINARY> where nulls are preserved. The JSON strings in the input must be valid.
- shred_
variant - Shreds the input binary variant using a target shredding schema derived from the requested data type.
- unshred_
variant - Removes all (nested) typed_value columns from a VariantArray by converting them back to binary variant and merging the resulting values back into the value column.
- variant_
get - Returns an array with the specified path extracted from the variant values.
- variant_
to_ json - Transform a batch of Variant represented as STRUCT<metadata: BINARY, value: BINARY> to a batch of JSON strings where nulls are preserved. The JSON strings in the input must be valid.