Crate parquet_variant_compute

VariantArray and compute kernels for the Variant Binary Encoding from Apache Parquet.

Main APIs

• VariantArray : Represents an array of Variant values.
• VariantArrayBuilder: For building VariantArray
• batch_json_string_to_variant: Function to convert a batch of JSON strings to a VariantArray.
• batch_variant_to_json_string: Function to convert a VariantArray to a batch of JSON strings.
• cast_to_variant: Module to cast other Arrow arrays to VariantArray.
• variant_get: Module to get values from a VariantArray using a specified VariantPath

🚧 Work In Progress

This crate is under active development and is not yet ready for production use. If you are interested in helping, you can find more information on the GitHub Variant issue

Modules

cast_to_variant

from_json 🔒

Module for transforming a batch of JSON strings into a batch of Variants represented as STRUCT<metadata: BINARY, value: BINARY>

to_json 🔒

Module for transforming a batch of Variants represented as STRUCT<metadata: BINARY, value: BINARY> into a batch of JSON strings.

variant_array 🔒

VariantArray implementation

variant_array_builder 🔒

VariantArrayBuilder implementation

variant_get

Structs

VariantArray

An array of Parquet [Variant] values

VariantArrayBuilder

A builder for VariantArray

VariantArrayVariantBuilder

A VariantBuilderExt that writes directly to the buffers of a VariantArrayBuilder.

Enums

ShreddingState

Represents the shredding state of a VariantArray

Functions

batch_json_string_to_variant

Parse a batch of JSON strings into a batch of Variants represented as STRUCT<metadata: BINARY, value: BINARY> where nulls are preserved. The JSON strings in the input must be valid.

batch_variant_to_json_string

Transform a batch of Variant represented as STRUCT<metadata: BINARY, value: BINARY> to a batch of JSON strings where nulls are preserved. The JSON strings in the input must be valid.