Struct GenericByteViewBuilder

pub struct GenericByteViewBuilder<T>where
    T: ByteViewType + ?Sized,{
    views_builder: BufferBuilder<u128>,
    null_buffer_builder: NullBufferBuilder,
    completed: Vec<Buffer>,
    in_progress: Vec<u8>,
    block_size: BlockSizeGrowthStrategy,
    string_tracker: Option<(HashTable<usize>, RandomState)>,
    phantom: PhantomData<T>,
}

Expand description

A builder for GenericByteViewArray

A GenericByteViewArray consists of a list of data blocks containing string data, and a list of views into those buffers.

See examples on StringViewBuilder and BinaryViewBuilder

This builder can be used in two ways

§Append Values

To avoid bump allocating, this builder allocates data in fixed size blocks, configurable using GenericByteViewBuilder::with_fixed_block_size. GenericByteViewBuilder::append_value writes values larger than 12 bytes to the current in-progress block, with values smaller than 12 bytes inlined into the views. If a value is appended that will not fit in the in-progress block, it will be closed, and a new block of sufficient size allocated

§Append Views

Some use-cases may wish to reuse an existing allocation containing string data, for example, when parsing data from a parquet data page. In such a case entire blocks can be appended using GenericByteViewBuilder::append_block and then views into this block appended using GenericByteViewBuilder::try_append_view

Fields§

§views_builder: BufferBuilder<u128>§null_buffer_builder: NullBufferBuilder§completed: Vec<Buffer>§in_progress: Vec<u8>§block_size: BlockSizeGrowthStrategy§string_tracker: Option<(HashTable<usize>, RandomState)>§phantom: PhantomData<T>

Implementations§

§

impl<T> GenericByteViewBuilder<T>
where T: ByteViewType + ?Sized,

pub fn new() -> GenericByteViewBuilder<T>

Creates a new GenericByteViewBuilder.

pub fn with_capacity(capacity: usize) -> GenericByteViewBuilder<T>

Creates a new GenericByteViewBuilder with space for capacity string values.

pub fn with_fixed_block_size(self, block_size: u32) -> GenericByteViewBuilder<T>

Set a fixed buffer size for variable length strings

The block size is the size of the buffer used to store values greater than 12 bytes. The builder allocates new buffers when the current buffer is full.

By default the builder balances buffer size and buffer count by growing buffer size exponentially from 8KB up to 2MB. The first buffer allocated is 8KB, then 16KB, then 32KB, etc up to 2MB.

If this method is used, any new buffers allocated are
exactly this size. This can be useful for advanced users that want to control the memory usage and buffer count.

See https://github.com/apache/arrow-rs/issues/6094 for more details on the implications.

pub fn with_block_size(self, block_size: u32) -> GenericByteViewBuilder<T>

👎Deprecated: Use with_fixed_block_size instead

Override the size of buffers to allocate for holding string data Use with_fixed_block_size instead.

pub fn with_deduplicate_strings(self) -> GenericByteViewBuilder<T>

Deduplicate strings while building the array

This will potentially decrease the memory usage if the array have repeated strings It will also increase the time to build the array as it needs to hash the strings

pub fn append_block(&mut self, buffer: Buffer) -> u32

Append a new data block returning the new block offset

Note: this will first flush any in-progress block

This allows appending views from blocks added using Self::append_block. See Self::append_value for appending individual values

let mut builder = StringViewBuilder::new();

let block = builder.append_block(b"helloworldbingobongo".into());

builder.try_append_view(block, 0, 5).unwrap();
builder.try_append_view(block, 5, 5).unwrap();
builder.try_append_view(block, 10, 5).unwrap();
builder.try_append_view(block, 15, 5).unwrap();
builder.try_append_view(block, 0, 15).unwrap();
let array = builder.finish();

let actual: Vec<_> = array.iter().flatten().collect();
let expected = &["hello", "world", "bingo", "bongo", "helloworldbingo"];
assert_eq!(actual, expected);

pub unsafe fn append_view_unchecked( &mut self, block: u32, offset: u32, len: u32, )

Append a view of the given block, offset and length

§Safety

(1) The block must have been added using Self::append_block (2) The range offset..offset+length must be within the bounds of the block (3) The data in the block must be valid of type T

pub fn try_append_view( &mut self, block: u32, offset: u32, len: u32, ) -> Result<(), ArrowError>

Try to append a view of the given block, offset and length

See Self::append_block

pub fn get_value(&self, index: usize) -> &[u8] ⓘ

Returns the value at the given index Useful if we want to know what value has been inserted to the builder The index has to be smaller than self.len(), otherwise it will panic