Skip to main content

Crate parquet_variant

Crate parquet_variant 

Source
Expand description

Implementation of Variant Binary Encoding from Apache Parquet.

Β§Main APIs

  • Variant: Represents a variant value, which can be an object, list, or primitive.
  • VariantBuilder: For building Variant values.

§🚧 Work In Progress

This crate is under active development and is not yet ready for production use. If you are interested in helping, you can find more information on the GitHub Variant issue

Β§Platform Support

Only little-endian platforms are officially supported and tested in CI. Big-endian platforms are not tested in CI and may not work correctly. Fixes for big-endian platforms are welcome and handled on a best-effort basis, but compatibility is not guaranteed.

Re-exportsΒ§

pub use path::VariantPath;
pub use path::VariantPathElement;
pub use self::decimal::VariantDecimal4;
pub use self::decimal::VariantDecimal8;
pub use self::decimal::VariantDecimal16;
pub use self::decimal::VariantDecimalType;
pub use self::list::VariantList;
pub use self::metadata::EMPTY_VARIANT_METADATA;
pub use self::metadata::EMPTY_VARIANT_METADATA_BYTES;
pub use self::metadata::VARIANT_SPEC_VERSION;
pub use self::metadata::VariantMetadata;
pub use self::object::VariantObject;

ModulesΒ§

builder πŸ”’
decimal
decoder πŸ”’
list
metadata
object
path πŸ”’
utils πŸ”’
variant πŸ”’

MacrosΒ§

variant_append_value
Macro to generate the match statement for each append_variant, try_append_variant, and append_variant_bytes – they each have slightly different handling for object and list handling.

StructsΒ§

HexString
InvalidVariant
ListBuilder
A builder for creating Variant::List values.
ListState
Internal state for list building
ObjectBuilder
A builder for creating Variant::Object values.
ObjectFieldBuilder
A VariantBuilderExt that inserts a new field into a variant object.
ObjectHeaderWriter
ObjectState
Internal state for object building
ParentState
Tracks information needed to correctly finalize a nested builder.
ReadOnlyMetadataBuilder
A metadata builder that cannot register new field names, and merely returns the field id associated with a known field name. This is useful for variant unshredding operations, where the metadata column is fixed and – per variant shredding spec – already contains all field names from the typed_value column. It is also useful when projecting a subset of fields from a variant object value, since the bytes can be copied across directly without re-encoding their field ids.
ShortString
A Variant ShortString
Uuid
A Universally Unique Identifier (UUID).
ValueBuilder
Wrapper around a Vec<u8> that provides methods for appending primitive values, variant types, and metadata.
VariantBuilder
Top level builder for Variant values
VariantType
Arrow Variant [ExtensionType].
WritableMetadataBuilder
Builder for constructing metadata for Variant values.
f16
A 16-bit floating point type implementing the IEEE 754-2008 standard binary16 a.k.a β€œhalf” format.

EnumsΒ§

Variant
Represents a Parquet Variant

ConstantsΒ§

BASIC_TYPE_BITS
EMPTY_BUILDER_MSG
MAX_NESTING_DEPTH
The maximum number of nested objects and arrays a Variant may contain.
MAX_SHORT_STRING_BYTES
ONE_TOP_LEVEL_VALUE_MSG
UNIX_EPOCH_DATE

TraitsΒ§

BuilderSpecificState
A trait for managing state specific to different builder types.
MetadataBuilder
A trait for building variant metadata dictionaries, to be used in conjunction with a ValueBuilder. The trait provides methods for managing field names and their IDs, as well as rolling back a failed builder operation that might have created new field ids.
VariantBuilderExt
Extends VariantBuilder to help building nested Variants

FunctionsΒ§

append_packed_u32
Append value_size bytes of given value into dest.
append_packed_u32
array_header
empty_builder_error
int_size
object_header
primitive_header
short_string_header
top_level_value_error
top_level_value_panic
write_offset
Write little-endian integer to buffer