arrow_array::builder::generic_bytes_view_builder

Struct GenericByteViewBuilder

Source
pub struct GenericByteViewBuilder<T: ByteViewType + ?Sized> {
    views_builder: BufferBuilder<u128>,
    null_buffer_builder: NullBufferBuilder,
    completed: Vec<Buffer>,
    in_progress: Vec<u8>,
    block_size: BlockSizeGrowthStrategy,
    string_tracker: Option<(HashTable<usize>, RandomState)>,
    phantom: PhantomData<T>,
}
Expand description

A builder for GenericByteViewArray

A GenericByteViewArray consists of a list of data blocks containing string data, and a list of views into those buffers.

See examples on StringViewBuilder and BinaryViewBuilder

This builder can be used in two ways

§Append Values

To avoid bump allocating, this builder allocates data in fixed size blocks, configurable using GenericByteViewBuilder::with_fixed_block_size. GenericByteViewBuilder::append_value writes values larger than 12 bytes to the current in-progress block, with values smaller than 12 bytes inlined into the views. If a value is appended that will not fit in the in-progress block, it will be closed, and a new block of sufficient size allocated

§Append Views

Some use-cases may wish to reuse an existing allocation containing string data, for example, when parsing data from a parquet data page. In such a case entire blocks can be appended using GenericByteViewBuilder::append_block and then views into this block appended using GenericByteViewBuilder::try_append_view

Fields§

§views_builder: BufferBuilder<u128>§null_buffer_builder: NullBufferBuilder§completed: Vec<Buffer>§in_progress: Vec<u8>§block_size: BlockSizeGrowthStrategy§string_tracker: Option<(HashTable<usize>, RandomState)>

Some if deduplicating strings map <string hash> -> <index to the views>

§phantom: PhantomData<T>

Implementations§

Source§

impl<T: ByteViewType + ?Sized> GenericByteViewBuilder<T>

Source

pub fn new() -> Self

Creates a new GenericByteViewBuilder.

Source

pub fn with_capacity(capacity: usize) -> Self

Creates a new GenericByteViewBuilder with space for capacity string values.

Source

pub fn with_fixed_block_size(self, block_size: u32) -> Self

Set a fixed buffer size for variable length strings

The block size is the size of the buffer used to store values greater than 12 bytes. The builder allocates new buffers when the current buffer is full.

By default the builder balances buffer size and buffer count by growing buffer size exponentially from 8KB up to 2MB. The first buffer allocated is 8KB, then 16KB, then 32KB, etc up to 2MB.

If this method is used, any new buffers allocated are
exactly this size. This can be useful for advanced users that want to control the memory usage and buffer count.

See https://github.com/apache/arrow-rs/issues/6094 for more details on the implications.

Source

pub fn with_block_size(self, block_size: u32) -> Self

👎Deprecated: Use with_fixed_block_size instead

Override the size of buffers to allocate for holding string data Use with_fixed_block_size instead.

Source

pub fn with_deduplicate_strings(self) -> Self

Deduplicate strings while building the array

This will potentially decrease the memory usage if the array have repeated strings It will also increase the time to build the array as it needs to hash the strings

Source

pub fn append_block(&mut self, buffer: Buffer) -> u32

Append a new data block returning the new block offset

Note: this will first flush any in-progress block

This allows appending views from blocks added using Self::append_block. See Self::append_value for appending individual values

let mut builder = StringViewBuilder::new();

let block = builder.append_block(b"helloworldbingobongo".into());

builder.try_append_view(block, 0, 5).unwrap();
builder.try_append_view(block, 5, 5).unwrap();
builder.try_append_view(block, 10, 5).unwrap();
builder.try_append_view(block, 15, 5).unwrap();
builder.try_append_view(block, 0, 15).unwrap();
let array = builder.finish();

let actual: Vec<_> = array.iter().flatten().collect();
let expected = &["hello", "world", "bingo", "bongo", "helloworldbingo"];
assert_eq!(actual, expected);
Source

pub unsafe fn append_view_unchecked( &mut self, block: u32, offset: u32, len: u32, )

Append a view of the given block, offset and length

§Safety

(1) The block must have been added using Self::append_block (2) The range offset..offset+length must be within the bounds of the block (3) The data in the block must be valid of type T

Source

pub fn try_append_view( &mut self, block: u32, offset: u32, len: u32, ) -> Result<(), ArrowError>

Try to append a view of the given block, offset and length

See Self::append_block

Source

fn flush_in_progress(&mut self)

Flushes the in progress block if any

Source

fn push_completed(&mut self, block: Buffer)

Append a block to self.completed, checking for overflow

Source

pub fn get_value(&self, index: usize) -> &[u8]

Returns the value at the given index Useful if we want to know what value has been inserted to the builder The index has to be smaller than self.len(), otherwise it will panic

Source

pub fn append_value(&mut self, value: impl AsRef<T::Native>)

Appends a value into the builder

§Panics

Panics if

  • String buffer count exceeds u32::MAX
  • String length exceeds u32::MAX
Source

pub fn append_option(&mut self, value: Option<impl AsRef<T::Native>>)

Append an Option value into the builder

Source

pub fn append_null(&mut self)

Append a null value into the builder

Source

pub fn finish(&mut self) -> GenericByteViewArray<T>

Builds the GenericByteViewArray and reset this builder

Source

pub fn finish_cloned(&self) -> GenericByteViewArray<T>

Builds the GenericByteViewArray without resetting the builder

Source

pub fn validity_slice(&self) -> Option<&[u8]>

Returns the current null buffer as a slice

Source

pub fn allocated_size(&self) -> usize

Return the allocated size of this builder in bytes, useful for memory accounting.

Trait Implementations§

Source§

impl<T: ByteViewType + ?Sized> ArrayBuilder for GenericByteViewBuilder<T>

Source§

fn len(&self) -> usize

Returns the number of array slots in the builder
Source§

fn finish(&mut self) -> ArrayRef

Builds the array
Source§

fn finish_cloned(&self) -> ArrayRef

Builds the array without resetting the underlying builder.
Source§

fn as_any(&self) -> &dyn Any

Returns the builder as a non-mutable Any reference. Read more
Source§

fn as_any_mut(&mut self) -> &mut dyn Any

Returns the builder as a mutable Any reference. Read more
Source§

fn into_box_any(self: Box<Self>) -> Box<dyn Any>

Returns the boxed builder as a box of Any.
Source§

fn is_empty(&self) -> bool

Returns whether number of array slots is zero
Source§

impl<T: ByteViewType + ?Sized> Debug for GenericByteViewBuilder<T>

Source§

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter. Read more
Source§

impl<T: ByteViewType + ?Sized> Default for GenericByteViewBuilder<T>

Source§

fn default() -> Self

Returns the “default value” for a type. Read more
Source§

impl<T: ByteViewType + ?Sized, V: AsRef<T::Native>> Extend<Option<V>> for GenericByteViewBuilder<T>

Source§

fn extend<I: IntoIterator<Item = Option<V>>>(&mut self, iter: I)

Extends a collection with the contents of an iterator. Read more
Source§

fn extend_one(&mut self, item: A)

🔬This is a nightly-only experimental API. (extend_one)
Extends a collection with exactly one element.
Source§

fn extend_reserve(&mut self, additional: usize)

🔬This is a nightly-only experimental API. (extend_one)
Reserves capacity in a collection for the given number of additional elements. Read more

Auto Trait Implementations§

§

impl<T> Freeze for GenericByteViewBuilder<T>
where T: ?Sized,

§

impl<T> RefUnwindSafe for GenericByteViewBuilder<T>
where T: RefUnwindSafe + ?Sized,

§

impl<T> Send for GenericByteViewBuilder<T>
where T: ?Sized,

§

impl<T> Sync for GenericByteViewBuilder<T>
where T: ?Sized,

§

impl<T> Unpin for GenericByteViewBuilder<T>
where T: Unpin + ?Sized,

§

impl<T> UnwindSafe for GenericByteViewBuilder<T>
where T: UnwindSafe + ?Sized,

Blanket Implementations§

Source§

impl<T> Any for T
where T: 'static + ?Sized,

Source§

fn type_id(&self) -> TypeId

Gets the TypeId of self. Read more
Source§

impl<T> Borrow<T> for T
where T: ?Sized,

Source§

fn borrow(&self) -> &T

Immutably borrows from an owned value. Read more
Source§

impl<T> BorrowMut<T> for T
where T: ?Sized,

Source§

fn borrow_mut(&mut self) -> &mut T

Mutably borrows from an owned value. Read more
Source§

impl<T> From<T> for T

Source§

fn from(t: T) -> T

Returns the argument unchanged.

Source§

impl<T, U> Into<U> for T
where U: From<T>,

Source§

fn into(self) -> U

Calls U::from(self).

That is, this conversion is whatever the implementation of From<T> for U chooses to do.

Source§

impl<T, U> TryFrom<U> for T
where U: Into<T>,

Source§

type Error = Infallible

The type returned in the event of a conversion error.
Source§

fn try_from(value: U) -> Result<T, <T as TryFrom<U>>::Error>

Performs the conversion.
Source§

impl<T, U> TryInto<U> for T
where U: TryFrom<T>,

Source§

type Error = <U as TryFrom<T>>::Error

The type returned in the event of a conversion error.
Source§

fn try_into(self) -> Result<U, <U as TryFrom<T>>::Error>

Performs the conversion.
§

impl<T> Allocation for T
where T: RefUnwindSafe + Send + Sync,