Struct OffsetBuffer

pub struct OffsetBuffer<I: OffsetSizeTrait> {
    pub offsets: Vec<I>,
    pub values: Vec<u8>,
}

A buffer of variable-sized byte arrays that can be converted into a corresponding [ArrayRef]

Fields

offsets: Vec<I>

values: Vec<u8>

Implementations

impl<I: OffsetSizeTrait> OffsetBuffer<I>

pub fn len(&self) -> usize

Returns the number of byte arrays in this buffer

pub fn is_empty(&self) -> bool

pub fn try_push(&mut self, data: &[u8], validate_utf8: bool) -> Result<()>

If validate_utf8 this verifies that the first character of data is the start of a UTF-8 codepoint

Note: This does not verify that the entirety of data is valid UTF-8. This should be done by calling Self::check_valid_utf8 after all data has been written

pub fn extend_from_dictionary<K: ArrowNativeType, V: ArrowNativeType>( &mut self, keys: &[K], dict_offsets: &[V], dict_values: &[u8], ) -> Result<()>

Extends this buffer with a list of keys

For each value key in keys this will insert &dict_values[dict_offsets[key]..dict_offsets[key+1]]

Note: This will validate offsets are valid

pub fn check_valid_utf8(&self, start_offset: usize) -> Result<()>

Validates that &self.values[start_offset..] is a valid UTF-8 sequence

This MUST be combined with validating that the offsets start on a character boundary, otherwise it would be possible for the values array to be a valid UTF-8 sequence, but not the individual string slices it contains

Self::try_push can perform this validation check on insertion

pub fn into_array( self, null_buffer: Option<Buffer>, data_type: ArrowType, ) -> ArrayRef

Converts this into an [ArrayRef] with the provided data_type and null_buffer

Trait Implementations

impl<I: Debug + OffsetSizeTrait> Debug for OffsetBuffer<I>

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl<I: OffsetSizeTrait> Default for OffsetBuffer<I>

fn default() -> Self

Returns the “default value” for a type.

impl<I: OffsetSizeTrait> ValuesBuffer for OffsetBuffer<I>

fn pad_nulls( &mut self, read_offset: usize, values_read: usize, levels_read: usize, valid_mask: &[u8], )

If a column contains nulls, more level data may be read than value data, as null values are not encoded. Therefore, first the levels data is read, the null count determined, and then the corresponding number of values read to a ValuesBuffer.