Expand description
Low-level buffer abstractions for Apache Arrow Rust
§Byte Storage abstractions
MutableBuffer: Raw memory buffer that can be mutated and grownBuffer: Immutable buffer that is shared across threads
§Typed Abstractions
There are also several wrappers over Buffer with methods for
easier manipulation:
BooleanBuffer: Bitmasks (buffer of packed bits)NullBuffer: Arrow null (validity) bitmaps (BooleanBufferwith extra utilities)ScalarBuffer<T>: Typed buffer for primitive types (e.g.,i32,f64)OffsetBuffer<O>: Offsets used in variable-length types (e.g., strings, lists)RunEndBuffer<E>: Run-ends used in run-encoded data
§Platform Support
Only little-endian platforms are officially supported and tested in CI. Big-endian platforms are not tested in CI and may not work correctly. Fixes for big-endian platforms are welcome and handled on a best-effort basis, but compatibility is not guaranteed.
Re-exports§
pub use error::OverflowError;pub use error::SliceOutOfBoundsError;pub use bigint::i256;pub use buffer::*;pub use builder::*;
Modules§
- alloc
- Defines the low-level
AllocationAPI for shared memory regions - arith 🔒
- bigint 🔒
- bit_
chunk_ iterator - Types for iterating over bitmasks in 64-bit chunks
- bit_
iterator - Types for iterating over packed bitmasks
- bit_
mask - Utils for working with packed bit masks
- bit_
util - Utils for working with bits
- buffer
- Types of shared memory region
- builder
- Buffer builders
- bytes 🔒
- This module contains an implementation of a contiguous immutable memory region that knows
how to de-allocate itself,
Bytes. Note that this is a low-level functionality of this crate. - divisor 🔒
- error 🔒
- Errors returned by
arrow-buffer. - interval 🔒
- native 🔒
- pool 🔒
- This module contains traits for memory pool traits and an implementation for tracking memory usage.
- private
- util 🔒
Macros§
- divisor_
enum - n_
bit_ divisor - A divisor with an N-bit multiplier, which computes the quotient as
MULUH(m, n') >> shiftforn'from$body.UpandDowndiffer only inn', and the macro keeps them from duplicating the rest.|n| ...looks like a closure but is macro syntax for the expression, because aconst fncan’t call a closure - native_
float - native_
integer - signed_
divisor - unsigned_
divisor
Structs§
- AddIndicator
Divisor I32 - A
i32divisor whose magnitude isn’t a power of two, seeDivisorI32 - AddIndicator
Divisor I64 - A
i64divisor whose magnitude isn’t a power of two, seeDivisorI64 - AddIndicator
Divisor U32 - A
u32divisor that isn’t a power of two, seeDivisorU32 - AddIndicator
Divisor U64 - A
u64divisor that isn’t a power of two, seeDivisorU64 - Down
Divisor U32 - A
u32divisor whose rounded-up multiplier isn’t exact in N bits, seeDivisorU32 - Interval
DayTime - Value of an IntervalDayTime array
- Interval
Month DayNano - Value of an IntervalMonthDayNano array
- Power
OfTwo Divisor I32 - A
i32divisor whose magnitude is a power of two, seeDivisorI32 - Power
OfTwo Divisor I64 - A
i64divisor whose magnitude is a power of two, seeDivisorI64 - Power
OfTwo Divisor U32 - A
u32divisor that is a power of two, seeDivisorU32 - Power
OfTwo Divisor U64 - A
u64divisor that is a power of two, seeDivisorU64 - Tracked
Reservation - This is a wrapper for the reservation so we can standardize on changing and avoid race conditions in memory accounting
- Tracker
- Tracking
Memory Pool - A simple
MemoryPoolthat reports the total memory usage - UpDivisor
U32 - A
u32divisor whose rounded-up multiplier is exact in N bits, seeDivisorU32 - UpDivisor
U64 - A
u64divisor whose rounded-up multiplier is exact in N bits, seeDivisorU64
Enums§
- Divisor
I32 - A
i32divisor, precomputed once to divide many values by it - Divisor
I64 - A
i64divisor, precomputed once to divide many values by it - Divisor
U32 - A
u32divisor, precomputed once to divide many values by it - Divisor
U64 - A
u64divisor, precomputed once to divide many values by it
Constants§
- CHEAP_
SATURATION - Whether a saturating increment on
u32lanes is cheap: oneuqaddon aarch64, andpminudfrom SSE4.1 and a subtract on x86. Baseline x86-64 has no unsigned 32-bit min and emulates it with compares, so there the saturating increment ofDivisorU32::Downcosts more than the add and halve ofDivisorU32::AddIndicator
Traits§
- Arrow
Native Type - Trait expressing a Rust type that has the same in-memory representation as Arrow.
- Memory
Pool - A pool of memory that can be reserved and released.
- Memory
Reservation - A memory reservation within a
MemoryPoolthat is freed on drop - ToByte
Slice - Allows conversion from supported Arrow types to a byte slice.
Functions§
- lock_
reservation - Lock a memory reservation, recovering from a poisoned lock.