Enum Encoding

pub enum Encoding {
    PLAIN,
    PLAIN_DICTIONARY,
    RLE,
    BIT_PACKED,
    DELTA_BINARY_PACKED,
    DELTA_LENGTH_BYTE_ARRAY,
    DELTA_BYTE_ARRAY,
    RLE_DICTIONARY,
    BYTE_STREAM_SPLIT,
}

Encodings supported by Parquet.

Not all encodings are valid for all types. These enums are also used to specify the encoding of definition and repetition levels.

By default this crate uses Encoding::PLAIN, Encoding::RLE, and Encoding::RLE_DICTIONARY. These provide very good encode and decode performance, whilst yielding reasonable storage efficiency and being supported by all major parquet readers.

The delta encodings are also supported and will be used if a newer WriterVersion is configured, however, it should be noted that these sacrifice encode and decode performance for improved storage efficiency. This performance regression is particularly pronounced in the case of record skipping as occurs during predicate push-down. It is recommended users assess the performance impact when evaluating these encodings.

Variants

PLAIN

Default byte encoding.

• BOOLEAN - 1 bit per value, 0 is false; 1 is true.
• INT32 - 4 bytes per value, stored as little-endian.
• INT64 - 8 bytes per value, stored as little-endian.
• FLOAT - 4 bytes per value, stored as little-endian.
• DOUBLE - 8 bytes per value, stored as little-endian.
• BYTE_ARRAY - 4 byte length stored as little endian, followed by bytes.
• FIXED_LEN_BYTE_ARRAY - just the bytes are stored.

PLAIN_DICTIONARY

Deprecated dictionary encoding.

The values in the dictionary are encoded using PLAIN encoding. Since it is deprecated, RLE_DICTIONARY encoding is used for a data page, and PLAIN encoding is used for dictionary page.

RLE

Group packed run length encoding.

Usable for definition/repetition levels encoding and boolean values.

BIT_PACKED

👎Deprecated since 51.0.0: Please see documentation for compatibility issues and use the RLE/bit-packing hybrid encoding instead

Deprecated Bit-packed encoding.

This can only be used if the data has a known max width. Usable for definition/repetition levels encoding.

There are compatibility issues with files using this encoding. The parquet standard specifies the bits to be packed starting from the most-significant bit, several implementations do not follow this bit order. Several other implementations also have issues reading this encoding because of incorrect assumptions about the length of the encoded data.

The RLE/bit-packing hybrid is more cpu and memory efficient and should be used instead.

DELTA_BINARY_PACKED

Delta encoding for integers, either INT32 or INT64.

Works best on sorted data.

DELTA_LENGTH_BYTE_ARRAY

Encoding for byte arrays to separate the length values and the data.

The lengths are encoded using DELTA_BINARY_PACKED encoding.

DELTA_BYTE_ARRAY

Incremental encoding for byte arrays.

Prefix lengths are encoded using DELTA_BINARY_PACKED encoding. Suffixes are stored using DELTA_LENGTH_BYTE_ARRAY encoding.

RLE_DICTIONARY

Dictionary encoding.

The ids are encoded using the RLE encoding.

BYTE_STREAM_SPLIT

Encoding for fixed-width data.

K byte-streams are created where K is the size in bytes of the data type. The individual bytes of a value are scattered to the corresponding stream and the streams are concatenated. This itself does not reduce the size of the data but can lead to better compression afterwards. Note that the use of this encoding with FIXED_LEN_BYTE_ARRAY(N) data may perform poorly for large values of N.

Trait Implementations

impl Clone for Encoding

fn clone(&self) -> Encoding

Returns a copy of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for Encoding

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Display for Encoding

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl From<Encoding> for Encoding

fn from(value: Encoding) -> Self

Converts to this type from the input type.

impl FromStr for Encoding

type Err = ParquetError

The associated error which can be returned from parsing.

fn from_str(s: &str) -> Result<Self, Self::Err>

Parses a string s to return a value of this type.

impl Hash for Encoding

fn hash<__H: Hasher>(&self, state: &mut __H)

Feeds this value into the given Hasher.

fn hash_slice<H>(data: &[Self], state: &mut H)

where H: Hasher, Self: Sized,

Feeds a slice of this type into the given Hasher.

impl HeapSize for Encoding

fn heap_size(&self) -> usize

Return the size of any bytes allocated on the heap by this object, including heap memory in those structures

impl Ord for Encoding

fn cmp(&self, other: &Encoding) -> Ordering

This method returns an Ordering between self and other.

fn max(self, other: Self) -> Self

where Self: Sized,

Compares and returns the maximum of two values.

fn min(self, other: Self) -> Self

where Self: Sized,

Compares and returns the minimum of two values.

fn clamp(self, min: Self, max: Self) -> Self

where Self: Sized,

Restrict a value to a certain interval.

impl PartialEq for Encoding

fn eq(&self, other: &Encoding) -> bool

Tests for self and other values to be equal, and is used by ==.

fn ne(&self, other: &Rhs) -> bool

Tests for !=. The default implementation is almost always sufficient, and should not be overridden without very good reason.

impl PartialOrd for Encoding

fn partial_cmp(&self, other: &Encoding) -> Option<Ordering>

This method returns an ordering between self and other values if one exists.

fn lt(&self, other: &Rhs) -> bool

Tests less than (for self and other) and is used by the < operator.

fn le(&self, other: &Rhs) -> bool

Tests less than or equal to (for self and other) and is used by the <= operator.

fn gt(&self, other: &Rhs) -> bool

Tests greater than (for self and other) and is used by the > operator.

fn ge(&self, other: &Rhs) -> bool

Tests greater than or equal to (for self and other) and is used by the >= operator.

impl TryFrom<Encoding> for Encoding

type Error = ParquetError

The type returned in the event of a conversion error.

fn try_from(value: Encoding) -> Result<Self>

Performs the conversion.

impl Copy for Encoding

impl Eq for Encoding

impl StructuralPartialEq for Encoding