Expand description
Dictionary utilities for Arrow arrays
Re-exportsยง
use std::collections::HashMap;๐use std::sync::Arc;๐use crate::filter::filter;๐use crate::interleave::interleave;๐use ahash::RandomState;๐use arrow_array::builder::BooleanBufferBuilder;๐use arrow_array::types::ArrowDictionaryKeyType;๐use arrow_array::types::ArrowPrimitiveType;๐use arrow_array::types::BinaryType;๐use arrow_array::types::ByteArrayType;๐use arrow_array::types::LargeBinaryType;๐use arrow_array::types::LargeUtf8Type;๐use arrow_array::types::Utf8Type;๐use arrow_array::AnyDictionaryArray;๐use arrow_array::Array;๐use arrow_array::ArrayRef;๐use arrow_array::ArrowNativeTypeOp;๐use arrow_array::BooleanArray;๐use arrow_array::DictionaryArray;๐use arrow_array::GenericByteArray;๐use arrow_array::PrimitiveArray;๐use arrow_array::downcast_dictionary_array;๐use arrow_array::cast::AsArray;๐use arrow_array::downcast_primitive;๐use arrow_buffer::ArrowNativeType;๐use arrow_buffer::BooleanBuffer;๐use arrow_buffer::ScalarBuffer;๐use arrow_buffer::ToByteSlice;๐use arrow_schema::ArrowError;๐use arrow_schema::DataType;๐
Macrosยง
Structsยง
- Interner ๐
- A best effort interner that maintains a fixed number of buckets and interns keys based on their hash value
- Merged
Dictionaries ๐
Enumsยง
- KeyMapping ๐
Constantsยง
Functionsยง
- bytes_
ptr_ ๐eq - Performs a cheap, pointer-based comparison of two byte array
- child_
values_ ๐ptr_ eq - Returns
trueif two child value arrays have the same layout and backing buffers. - compute_
referenced_ ๐keys - Returns the unique dictionary keys referenced by the selected rows.
- compute_
values_ ๐mask - Return a mask identifying the values that are referenced by keys in
dictionaryat the positions indicated byselection - garbage_
collect_ any_ dictionary - Equivalent to
garbage_collect_dictionarybut without requiring casting to a specific key type. - garbage_
collect_ dictionary - Garbage collects a [DictionaryArray] by removing unreferenced values.
- get_
selected_ ๐values - merge_
dictionary_ ๐values - Given an array of dictionaries and an optional key mask compute a values array
containing referenced values, along with mappings from the [
DictionaryArray] keys to the new keys within this values array. Best-effort will be made to ensure that the dictionary values are unique - merge_
dictionary_ ๐values_ with_ masks - Merges referenced entries from multiple dictionary values arrays.
- selected_
bytes ๐ - selected_
primitives_ ๐to_ bytes - should_
merge_ ๐dictionary_ values - A weak heuristic of whether to merge dictionary values that aims to only perform the expensive merge computation when it is likely to yield at least some return over the naive approach used by MutableArrayData
Type Aliasesยง
- Interner
Bucket ๐ - A single bucket in
Interner. - PtrEq ๐
- A type-erased function that compares two array for pointer equality