Enum Variant

Source

pub enum Variant<'m, 'v> {
Show 20 variants    Null,
    Int8(i8),
    Int16(i16),
    Int32(i32),
    Int64(i64),
    Date(NaiveDate),
    TimestampMicros(DateTime<Utc>),
    TimestampNtzMicros(NaiveDateTime),
    Decimal4(VariantDecimal4),
    Decimal8(VariantDecimal8),
    Decimal16(VariantDecimal16),
    Float(f32),
    Double(f64),
    BooleanTrue,
    BooleanFalse,
    Binary(&'v [u8]),
    String(&'v str),
    ShortString(ShortString<'v>),
    Object(VariantObject<'m, 'v>),
    List(VariantList<'m, 'v>),
}

Expand description

Represents a Parquet Variant

The lifetimes 'm and 'v are for metadata and value buffers, respectively.

§Background

The specification says:

The Variant Binary Encoding allows representation of semi-structured data (e.g. JSON) in a form that can be efficiently queried by path. The design is intended to allow efficient access to nested data even in the presence of very wide or deep structures.

Another motivation for the representation is that (aside from metadata) each nested Variant value is contiguous and self-contained. For example, in a Variant containing an Array of Variant values, the representation of an inner Variant value, when paired with the metadata of the full variant, is itself a valid Variant.

When stored in Parquet files, Variant fields can also be shredded. Shredding refers to extracting some elements of the variant into separate columns for more efficient extraction/filter pushdown. The Variant Shredding specification describes the details of shredding Variant values as typed Parquet columns.

A Variant represents a type that contains one of:

Primitive: A type and corresponding value (e.g. INT, STRING)
Array: An ordered list of Variant values
Object: An unordered collection of string/Variant pairs (i.e. key/value pairs). An object may not contain duplicate keys.

§Encoding

A Variant is encoded with 2 binary values, the value and the metadata. The metadata stores a header and an optional dictionary of field names which are referred to by offset in the value. The value is a binary representation of the actual data, and varies depending on the type.

§Design Goals

The design goals of the Rust API are as follows:

Speed / Zero copy access (no cloneing is required)
Safety
Follow standard Rust conventions

§Examples:

§Creating `Variant` from Rust Types

use parquet_variant::Variant;
// variants can be directly constructed
let variant = Variant::Int32(123);
// or constructed via `From` impls
assert_eq!(variant, Variant::from(123i32));

§Creating `Variant` from metadata and value

let metadata = [0x01, 0x00, 0x00];
let value = [0x09, 0x48, 0x49];
// parse the header metadata
assert_eq!(
  Variant::from("HI"),
  Variant::new(&metadata, &value)
);

§Using `Variant` values

// variants can be used in match statements like normal enums
match variant {
  Variant::Int32(i) => println!("Integer: {}", i),
  Variant::String(s) => println!("String: {}", s),
  _ => println!("Other variant"),
}

§Validation

Every instance of variant is either valid or invalid. depending on whether the underlying bytes are a valid encoding of a variant value (see below).

Instances produced by Self::try_new, Self::try_new_with_metadata, or Self::with_full_validation are fully validated. They always contain valid data, and infallible accesses such as iteration and indexing are panic-free. The validation cost is O(m + v) where m and v are the number of bytes in the metadata and value buffers, respectively.

Instances produced by Self::new and Self::new_with_metadata are unvalidated and so they may contain either valid or invalid data. Infallible accesses to variant objects and arrays, such as iteration and indexing will panic if the underlying bytes are invalid, and fallible alternatives are provided as panic-free alternatives. Self::with_full_validation can also be used to validate an unvalidated instance, if desired.

Unvalidated instances can be constructed in constant time. This can be useful if the caller knows the underlying bytes were already validated previously, or if the caller intends to perform a small number of (fallible) accesses to a large variant value.

A validated variant value guarantees that the associated metadata and all nested object and array values are valid. Primitive variant subtypes are always valid by construction.

§Safety

Even an invalid variant value is still safe to use in the Rust sense. Accessing it with infallible methods may cause panics but will never lead to undefined behavior.