Struct MutableArrayData

Source

pub struct MutableArrayData<'a> {
    arrays: Vec<&'a ArrayData>,
    data: _MutableArrayData<'a>,
    dictionary: Option<ArrayData>,
    variadic_data_buffers: Vec<Buffer>,
    extend_values: Vec<Box<dyn Fn(&mut _MutableArrayData<'_>, usize, usize, usize) -> Result<(), ArrowError> + 'a>>,
    extend_null_bits: Vec<Box<dyn Fn(&mut _MutableArrayData<'_>, usize, usize) + 'a>>,
    extend_nulls: Box<dyn Fn(&mut _MutableArrayData<'_>, usize) -> Result<(), ArrowError>>,
}

Expand description

Efficiently create an ArrayData from one or more existing ArrayDatas by copying chunks.

The main use case of this struct is to perform unary operations to arrays of arbitrary types, such as filter and take.

§Example

use arrow_buffer::Buffer;
use arrow_data::ArrayData;
use arrow_data::transform::MutableArrayData;
use arrow_schema::DataType;
fn i32_array(values: &[i32]) -> ArrayData {
  ArrayData::try_new(DataType::Int32, values.len(), None, 0, vec![Buffer::from_slice_ref(values)], vec![]).unwrap()
}
let arr1  = i32_array(&[1, 2, 3, 4, 5]);
let arr2  = i32_array(&[6, 7, 8, 9, 10]);
// Create a mutable array for copying values from arr1 and arr2, with a capacity for 6 elements
let capacity = 3 * std::mem::size_of::<i32>();
let mut mutable = MutableArrayData::new(vec![&arr1, &arr2], false, 10);
// Copy the first 3 elements from arr1
mutable.extend(0, 0, 3);
// Copy the last 3 elements from arr2
mutable.extend(1, 2, 5);
// Complete the MutableArrayData into a new ArrayData
let frozen = mutable.freeze();
assert_eq!(frozen, i32_array(&[1, 2, 3, 8, 9, 10]));

Fields§

§arrays: Vec<&'a ArrayData>

Input arrays: the data being read FROM.

Note all actual reads of the arrays go through the closures for extending values and nulls; these references are only kept for bounds checking.

§data: _MutableArrayData<'a>

In progress output array: The data being written TO

Note these fields are in a separate struct, _MutableArrayData, as they cannot be in MutableArrayData itself due to mutability invariants (interior mutability): MutableArrayData contains a function that can only mutate _MutableArrayData, not MutableArrayData itself

§dictionary: Option<ArrayData>

The child data of the Array in Dictionary arrays.

This is not stored in _MutableArrayData because these values are constant and only needed at the end, when freezing _MutableArrayData.

§variadic_data_buffers: Vec<Buffer>

Variadic data buffers referenced by views.

Note this this is not stored in _MutableArrayData because these values are constant and only needed at the end, when freezing _MutableArrayData

§extend_values: Vec<Box<dyn Fn(&mut _MutableArrayData<'_>, usize, usize, usize) -> Result<(), ArrowError> + 'a>>

function used to extend output array with values from input arrays.

This function’s lifetime is bound to the input arrays because it reads values from them.

§extend_null_bits: Vec<Box<dyn Fn(&mut _MutableArrayData<'_>, usize, usize) + 'a>>

function used to extend the output array with nulls from input arrays.

This function’s lifetime is bound to the input arrays because it reads nulls from it.

§extend_nulls: Box<dyn Fn(&mut _MutableArrayData<'_>, usize) -> Result<(), ArrowError>>

function used to extend the output array with null elements.

This function is independent of the arrays and therefore has no lifetime.

Implementations§

Source §

impl<'a> MutableArrayData<'a>

Source

pub fn new(arrays: Vec<&'a ArrayData>, use_nulls: bool, capacity: usize) -> Self

Returns a new MutableArrayData with capacity to capacity slots and specialized to create an ArrayData from multiple arrays.

§Arguments

arrays - the source arrays to copy from
use_nulls - a flag indicating whether the caller intends to call extend_nulls. Note: null-handling is enabled automatically if any source array contains nulls.
capacity - the preallocated capacity of the output array, in slots (number of elements)

if use_nulls is false and no source arrays contains nulls, calling MutableArrayData::extend_nulls or MutableArrayData::try_extend_nulls will panic.

Source

pub fn try_new( arrays: Vec<&'a ArrayData>, use_nulls: bool, capacity: usize, ) -> Result<Self, ArrowError>

Fallible variant of MutableArrayData::new.

Unlike MutableArrayData::new, this does not panic when merging dictionary arrays whose combined values would overflow the dictionary key type. Instead, it returns an error, letting callers (e.g. interleave / concat) surface it as a normal error.

Source

pub fn with_capacities( arrays: Vec<&'a ArrayData>, use_nulls: bool, capacities: Capacities, ) -> Self

Similar to MutableArrayData::new, but lets users define the preallocated capacities of the array with more granularity.

See MutableArrayData::new for more information on the arguments.

§Panics

if the given capacities don’t match the data type of arrays
if a Capacities variant is not yet supported
when merging dictionary arrays whose combined values overflow the dictionary key type — see MutableArrayData::try_with_capacities for a fallible variant

Source

pub fn try_with_capacities( arrays: Vec<&'a ArrayData>, use_nulls: bool, capacities: Capacities, ) -> Result<Self, ArrowError>

Fallible variant of MutableArrayData::with_capacities.

Returns an error instead of panicking when merging dictionary arrays whose combined values would overflow the dictionary key type. Still panics for other unsupported combinations (inconsistent input types, unsupported Capacities variants) as documented on MutableArrayData::with_capacities.

Source

pub fn try_extend( &mut self, index: usize, start: usize, end: usize, ) -> Result<(), ArrowError>

Extends the in progress array with a region of the input arrays, returning an error on overflow.

§Arguments

index - the index of array that you want to copy values from
start - the start index of the chunk (inclusive)
end - the end index of the chunk (exclusive)

§Errors

Returns an error if

index >= the number of source arrays,
start..end is not a valid range within the indexth array, or
offset arithmetic overflows the underlying integer type.

Source

pub fn extend(&mut self, index: usize, start: usize, end: usize)

👎Deprecated since 59.0.0:

Use try_extend which returns an error on overflow instead of panicking

Extends the in progress array with a region of the input arrays.

§Panics

This function panics if

index >= the number of source arrays,
start..end is not a valid range within the indexth array, or
the offset type overflows (e.g. more than 2 GiB in a StringArray).

Source

pub fn try_extend_nulls(&mut self, len: usize) -> Result<(), ArrowError>

Extends the in progress array with null elements, ignoring the input arrays, returning an error on overflow.

Prefer this over extend_nulls to handle cases where the run-end counter overflows (relevant for RunEndEncoded arrays).

§Errors

Returns an error if this MutableArrayData was not created with use_nulls and none of the source arrays are nullable, or if the run-end counter overflows.

Source