Struct InProgressByteViewArray

pub(crate) struct InProgressByteViewArray<B: ByteViewType> {
    source: Option<Source>,
    batch_size: usize,
    views: Vec<u128>,
    nulls: NullBufferBuilder,
    current: Option<Vec<u8>>,
    completed: Vec<Buffer>,
    buffer_source: BufferSource,
    _phantom: PhantomData<B>,
}

InProgressArray for StringViewArray and BinaryViewArray

This structure buffers the views and data buffers as they are copied from the source array, and then produces a new array when finish is called. It also handles “garbage collection” by copying strings to a new buffer when the source buffer is sparse (i.e. uses at least 2x more than the memory it needs).

Fields

source: Option<Source>

The source array and information

batch_size: usize

the target batch size (and thus size for views allocation)

views: Vec<u128>

The in progress views

nulls: NullBufferBuilder

In progress nulls

current: Option<Vec<u8>>

current buffer

completed: Vec<Buffer>

completed buffers

buffer_source: BufferSource

Allocates new buffers of increasing size as needed

_phantom: PhantomData<B>

Phantom so we can use the same struct for both StringViewArray and BinaryViewArray

Implementations

impl<B: ByteViewType> InProgressByteViewArray<B>

pub(crate) fn new(batch_size: usize) -> Self

fn ensure_capacity(&mut self)

Allocate space for output views and nulls if needed

This is done on write (when we know it is necessary) rather than eagerly to avoid allocations that are not used.

fn finish_current(&mut self)

Finishes in progress buffer, if any

fn append_views_and_update_buffer_index( &mut self, views: &[u128], buffers: &[Buffer], )

Append views to self.views, updating the buffer index if necessary

fn append_views_and_copy_strings( &mut self, views: &[u128], view_buffer_size: usize, buffers: &[Buffer], )

Append views to self.views, copying data from the buffers into self.buffers and updating the buffer index as necessary.

Arguments

• views - the views to append
• view_buffer_size - the total number of bytes pointed to by all views (used to allocate new buffers if needed)
• buffers - the buffers the reviews point to

fn append_views_and_copy_strings_inner( &mut self, views: &[u128], dst_buffer: Vec<u8>, buffers: &[Buffer], )

Append views to self.views, copying data from the buffers into dst_buffer, which is then set as self.current

Panics:

If self.current is Some

See append_views_and_copy_strings for more details

Trait Implementations

impl<B: ByteViewType> Debug for InProgressByteViewArray<B>

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl<B: ByteViewType> InProgressArray for InProgressByteViewArray<B>

fn set_source(&mut self, source: Option<ArrayRef>)

Set the source array.

fn copy_rows(&mut self, offset: usize, len: usize) -> Result<(), ArrowError>

Copy rows from the current source array into the in-progress array

fn finish(&mut self) -> Result<ArrayRef, ArrowError>

Finish the currently in-progress array and return it as an ArrayRef