Skip to main content

Module dictionary

Module dictionary 

Source
Expand description

Dictionary utilities for Arrow arrays

Re-exportsยง

use std::collections::HashMap; ๐Ÿ”’
use std::sync::Arc; ๐Ÿ”’
use crate::filter::filter; ๐Ÿ”’
use crate::interleave::interleave; ๐Ÿ”’
use ahash::RandomState; ๐Ÿ”’
use arrow_array::builder::BooleanBufferBuilder; ๐Ÿ”’
use arrow_array::types::ArrowDictionaryKeyType; ๐Ÿ”’
use arrow_array::types::ArrowPrimitiveType; ๐Ÿ”’
use arrow_array::types::BinaryType; ๐Ÿ”’
use arrow_array::types::ByteArrayType; ๐Ÿ”’
use arrow_array::types::LargeBinaryType; ๐Ÿ”’
use arrow_array::types::LargeUtf8Type; ๐Ÿ”’
use arrow_array::types::Utf8Type; ๐Ÿ”’
use arrow_array::AnyDictionaryArray; ๐Ÿ”’
use arrow_array::Array; ๐Ÿ”’
use arrow_array::ArrayRef; ๐Ÿ”’
use arrow_array::ArrowNativeTypeOp; ๐Ÿ”’
use arrow_array::BooleanArray; ๐Ÿ”’
use arrow_array::DictionaryArray; ๐Ÿ”’
use arrow_array::GenericByteArray; ๐Ÿ”’
use arrow_array::PrimitiveArray; ๐Ÿ”’
use arrow_array::downcast_dictionary_array; ๐Ÿ”’
use arrow_array::cast::AsArray; ๐Ÿ”’
use arrow_array::downcast_primitive; ๐Ÿ”’
use arrow_buffer::ArrowNativeType; ๐Ÿ”’
use arrow_buffer::BooleanBuffer; ๐Ÿ”’
use arrow_buffer::ScalarBuffer; ๐Ÿ”’
use arrow_buffer::ToByteSlice; ๐Ÿ”’
use arrow_schema::ArrowError; ๐Ÿ”’
use arrow_schema::DataType; ๐Ÿ”’

Macrosยง

selected_primitive_to_bytes_helper ๐Ÿ”’

Structsยง

Interner ๐Ÿ”’
A best effort interner that maintains a fixed number of buckets and interns keys based on their hash value
MergedDictionaries ๐Ÿ”’

Enumsยง

KeyMapping ๐Ÿ”’

Constantsยง

MAX_DENSE_MAPPING_VALUES_PER_ROW ๐Ÿ”’

Functionsยง

bytes_ptr_eq ๐Ÿ”’
Performs a cheap, pointer-based comparison of two byte array
child_values_ptr_eq ๐Ÿ”’
Returns true if two child value arrays have the same layout and backing buffers.
compute_referenced_keys ๐Ÿ”’
Returns the unique dictionary keys referenced by the selected rows.
compute_values_mask ๐Ÿ”’
Return a mask identifying the values that are referenced by keys in dictionary at the positions indicated by selection
garbage_collect_any_dictionary
Equivalent to garbage_collect_dictionary but without requiring casting to a specific key type.
garbage_collect_dictionary
Garbage collects a [DictionaryArray] by removing unreferenced values.
get_selected_values ๐Ÿ”’
merge_dictionary_values ๐Ÿ”’
Given an array of dictionaries and an optional key mask compute a values array containing referenced values, along with mappings from the [DictionaryArray] keys to the new keys within this values array. Best-effort will be made to ensure that the dictionary values are unique
merge_dictionary_values_with_masks ๐Ÿ”’
Merges referenced entries from multiple dictionary values arrays.
selected_bytes ๐Ÿ”’
selected_primitives_to_bytes ๐Ÿ”’
should_merge_dictionary_values ๐Ÿ”’
A weak heuristic of whether to merge dictionary values that aims to only perform the expensive merge computation when it is likely to yield at least some return over the naive approach used by MutableArrayData

Type Aliasesยง

InternerBucket ๐Ÿ”’
A single bucket in Interner.
PtrEq ๐Ÿ”’
A type-erased function that compares two array for pointer equality