Module dictionary

Dictionary utilities for Arrow arrays

Macros

masked_primitive_to_bytes_helper 🔒

Structs

Interner 🔒

A best effort interner that maintains a fixed number of buckets and interns keys based on their hash value

MergedDictionaries 🔒

Functions

bytes_ptr_eq 🔒

Performs a cheap, pointer-based comparison of two byte array

compute_values_mask 🔒

Return a mask identifying the values that are referenced by keys in dictionary at the positions indicated by selection

garbage_collect_any_dictionary

Equivalent to garbage_collect_dictionary but without requiring casting to a specific key type.

garbage_collect_dictionary

Garbage collects a [DictionaryArray] by removing unreferenced values.

get_masked_values 🔒

Return a Vec containing for each set index in mask, the index and byte value of that index

masked_bytes 🔒

Compute get_masked_values for a [GenericByteArray]

masked_primitives_to_bytes 🔒

Process primitive array values to bytes

merge_dictionary_values 🔒

Given an array of dictionaries and an optional key mask compute a values array containing referenced values, along with mappings from the [DictionaryArray] keys to the new keys within this values array. Best-effort will be made to ensure that the dictionary values are unique

should_merge_dictionary_values 🔒

A weak heuristic of whether to merge dictionary values that aims to only perform the expensive merge computation when it is likely to yield at least some return over the naive approach used by MutableArrayData

Type Aliases

InternerBucket 🔒

A single bucket in Interner.

PtrEq 🔒

A type-erased function that compares two array for pointer equality