Struct RunArray 

pub struct RunArray<R: RunEndIndexType> {
    data_type: DataType,
    run_ends: RunEndBuffer<R::Native>,
    values: ArrayRef,
}

An array of run-end encoded values.

This encoding is variation on run-length encoding (RLE) and is good for representing data containing the same values repeated consecutively.

A RunArray consists of a run_ends buffer and a values array of equivalent lengths. The run_ends buffer stores the indexes at which the run ends. The values array stores the corresponding value of each run. The below example illustrates how a logical array is represented by a RunArray:

┌ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─┐
  ┌─────────────────┐  ┌─────────┐       ┌─────────────────┐
│ │        A        │  │    2    │ │     │        A        │
  ├─────────────────┤  ├─────────┤       ├─────────────────┤
│ │        D        │  │    3    │ │     │        A        │    run length of 'A' = run_ends[0] - 0 = 2
  ├─────────────────┤  ├─────────┤       ├─────────────────┤
│ │        B        │  │    6    │ │     │        D        │    run length of 'D' = run_ends[1] - run_ends[0] = 1
  └─────────────────┘  └─────────┘       ├─────────────────┤
│        values          run_ends  │     │        B        │
                                         ├─────────────────┤
└ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─┘     │        B        │
                                         ├─────────────────┤
               RunArray                  │        B        │    run length of 'B' = run_ends[2] - run_ends[1] = 3
         offset = 0, len = 6             └─────────────────┘

                                            Logical array
                                               Contents

offset and len

The Array::len and Array::offset for a RunArray refer to the logical length, not the run_ends and values child arrays. In the example above the RunArray has length 6 even though run_ends and values each contain 3 elements.

The values in run_ends represent indexes of the unsliced logical array. This allows zero-copy RunArray::slice: run_ends and values are left untouched and only offset and len are adjusted.

A RunArray must satisfy the following invariants:

• every run end is greater than zero and run_ends is strictly increasing
• run_ends has no null values
• run_ends and values have the same length
• if len > 0, the last run end is greater than or equal to offset + len.

Note an empty RunArray (len == 0) is valid even when run_ends is empty or its last run end is less than offset. Note this is more permissive than the Arrow C++ implementation.

The example below shows the same array as above after calling slice(1, 3)

┌ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─┐
  ┌─────────────────┐  ┌─────────┐       ┌ ─ ─ ─ ─ ─ ─ ─ ─ ┐
│ │        A        │  │    2    │ │              A             logical index 0: before offset, not visible
  ├─────────────────┤  ├─────────┤       ┌─────────────────┐ ◄── offset = 1
│ │        D        │  │    3    │ │     │        A        │    logical index 1 < run_ends[0] = 2, so 'A'
  ├─────────────────┤  ├─────────┤       ├─────────────────┤
│ │        B        │  │    6    │ │     │        D        │    logical index 2 < run_ends[1] = 3, so 'D'
  └─────────────────┘  └─────────┘       ├─────────────────┤
│        values          run_ends  │     │        B        │    logical index 3 < run_ends[2] = 6, so 'B'
                                         └─────────────────┘ ◄── offset + len = 4
└ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─┘              B             logical index 4: after offset + len, not visible
                                         ├ ─ ─ ─ ─ ─ ─ ─ ─ ┤
               RunArray                           B             logical index 5: after offset + len, not visible
         offset = 1, len = 3             └ ─ ─ ─ ─ ─ ─ ─ ─ ┘

                                            Logical array
                                               Contents

Sliced child arrays

RunArray::slice doesn’t slice child arrays, but a RunArray may be built (for example from [ArrayData]) from child arrays that are already sliced. In that case each child’s own offset and len are respected, independently of the logical offset and len of the RunArray:

• Run i ends at run_ends.value(i), the ith (logical) element
• The value of run i is values.value(i), the ith (logical) element

Fields

data_type: DataType

run_ends: RunEndBuffer<R::Native>

values: ArrayRef

Implementations

impl<R: RunEndIndexType> RunArray<R>

pub fn logical_len(run_ends: &PrimitiveArray<R>) -> usize

Calculates the logical length of the array encoded by treating the run_ends array as if it were a [RunEndBuffer].

pub fn try_new( run_ends: &PrimitiveArray<R>, values: &dyn Array, ) -> Result<Self, ArrowError>

Attempts to create a RunArray using the given run_ends and values.

The resulting array is unsliced: its logical offset is 0 and its logical length is the last value of run_ends (or 0 if run_ends is empty).

Errors

Returns an error if the RunArray invariants are not satisfied:

• If run_ends and values have different lengths
• If run_ends has any null values
• If run_ends doesn’t consist of strictly increasing positive integers

pub unsafe fn new_unchecked( data_type: DataType, run_ends: RunEndBuffer<R::Native>, values: ArrayRef, ) -> Self

Create a new RunArray from the provided parts, without validation

Safety

Safe if Self::try_new would not error, that is if run_ends and values satisfy the RunArray invariants

pub fn into_parts(self) -> (DataType, RunEndBuffer<R::Native>, ArrayRef)

Deconstruct this array into its constituent parts

pub fn run_ends_field(&self) -> &FieldRef

The field that describes the run ends of this array.

pub fn values_field(&self) -> &FieldRef

The field that describes the values of this array.

pub fn run_ends(&self) -> &RunEndBuffer<R::Native>

Returns a reference to the [RunEndBuffer].

pub fn values(&self) -> &ArrayRef

Returns a reference to the values array.

Any slicing of this RunArray array is not applied to the returned values here and must be handled separately.

pub fn with_values(&self, values: ArrayRef) -> Self

Returns a new RunArray with the same run_ends and the supplied values.

Panics

Panics if values.len() does not equal self.values().len().

Example

// A RunArray logically representing ["a", "a", "b", "c", "c"]
let run_ends = Int32Array::from(vec![2, 3, 5]);
let values: ArrayRef = Arc::new(StringArray::from(vec!["a", "b", "c"]));
let run_array = RunArray::<Int32Type>::try_new(&run_ends, &values).unwrap();

// Swap in new values while keeping the same run pattern.
// The result logically represents ["x", "x", "y", "z", "z"].
let new_values: ArrayRef = Arc::new(StringArray::from(vec!["x", "y", "z"]));
let new_run_array = run_array.with_values(new_values);

assert_eq!(new_run_array.len(), 5);
assert_eq!(new_run_array.run_ends().values(), &[2, 3, 5]);

pub fn values_slice(&self) -> ArrayRef

Similar to values but accounts for logical slicing, returning only the values that are part of the logical slice of this array.

pub fn get_start_physical_index(&self) -> usize

Returns the physical index at which the array slice starts.

See [RunEndBuffer::get_start_physical_index].

pub fn get_end_physical_index(&self) -> usize

Returns the physical index at which the array slice ends.

See [RunEndBuffer::get_end_physical_index].

pub fn downcast<V: 'static>(&self) -> Option<TypedRunArray<'_, R, V>>

Downcast this RunArray to a TypedRunArray

use arrow_array::{Array, ArrayAccessor, RunArray, StringArray, types::Int32Type};

let orig = [Some("a"), Some("b"), None];
let run_array = RunArray::<Int32Type>::from_iter(orig);
let typed = run_array.downcast::<StringArray>().unwrap();
assert_eq!(typed.value(0), "a");
assert_eq!(typed.value(1), "b");
assert!(typed.values().is_null(2));

pub fn get_physical_index(&self, logical_index: usize) -> usize

Calls [RunEndBuffer::get_physical_index].

The result is arbitrary if logical_index >= self.len()

pub fn get_physical_indices<I>( &self, logical_indices: &[I], ) -> Result<Vec<usize>, ArrowError>

where I: ArrowNativeType,

Returns the physical indices corresponding to the provided logical indices.

See [RunEndBuffer::get_physical_indices] for more details.

pub fn slice(&self, offset: usize, length: usize) -> Self

Returns a zero-copy slice of this array with the indicated offset and length.

Panics

• Specified slice (offset + length) exceeds existing length

Trait Implementations

impl<R: RunEndIndexType> AnyRunEndArray for RunArray<R>

fn values(&self) -> &Arc<dyn Array>

Returns the values of this array.

fn with_values(&self, values: ArrayRef) -> ArrayRef

Returns a new run-end encoded array with the given values, preserving the existing run ends.

impl<T: RunEndIndexType> Array for RunArray<T>

SAFETY: Correctly implements the contract of Arrow Arrays

fn as_any(&self) -> &dyn Any

Returns the array as Any so that it can be downcasted to a specific implementation.

fn to_data(&self) -> ArrayData

Returns the underlying data of this array

fn into_data(self) -> ArrayData

Returns the underlying data of this array

fn data_type(&self) -> &DataType

Returns a reference to the [DataType] of this array.

fn slice(&self, offset: usize, length: usize) -> ArrayRef

Returns a zero-copy slice of this array with the indicated offset and length.

fn len(&self) -> usize

Returns the length (i.e., number of elements) of this array.

fn is_empty(&self) -> bool

Returns whether this array is empty.

fn shrink_to_fit(&mut self)

Shrinks the capacity of any exclusively owned buffer as much as possible

fn offset(&self) -> usize

Returns the offset into the underlying data used by this array(-slice). Note that the underlying data can be shared by many arrays. This defaults to 0.

fn nulls(&self) -> Option<&NullBuffer>

Returns the null buffer of this array if any.

fn logical_nulls(&self) -> Option<NullBuffer>

Returns a potentially computed [NullBuffer] that represents the logical null values of this array, if any.

fn is_nullable(&self) -> bool

Returns false if the array is guaranteed to not contain any logical nulls

fn get_buffer_memory_size(&self) -> usize

Returns the total number of bytes of memory pointed to by this array. The buffers store bytes in the Arrow memory format, and include the data as well as the validity map. Note that this does not always correspond to the exact memory usage of an array, since multiple arrays can share the same buffers or slices thereof.

fn get_array_memory_size(&self) -> usize

Returns the total number of bytes of memory occupied physically by this array. This value will always be greater than returned by get_buffer_memory_size() and includes the overhead of the data structures that contain the pointers to the various buffers.

fn claim(&self, pool: &dyn MemoryPool)

Claim memory used by this array in the provided memory pool.

fn is_null(&self, index: usize) -> bool

Returns whether the element at index is null according to Array::nulls

fn is_valid(&self, index: usize) -> bool

Returns whether the element at index is not null, the opposite of Self::is_null.

fn null_count(&self) -> usize

Returns the total number of physical null values in this array.

fn logical_null_count(&self) -> usize

Returns the total number of logical null values in this array.

impl<R: RunEndIndexType> Clone for RunArray<R>

fn clone(&self) -> Self

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl<R: RunEndIndexType> Debug for RunArray<R>

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl<R: RunEndIndexType> From<ArrayData> for RunArray<R>

Converts an [ArrayData] into a RunArray.

The len and offset of data become the logical length and offset of the RunArray; see the RunArray documentation for how they relate to the run ends.

The offset and len of the run_ends child are respected: only the len elements starting at offset are used as run ends. See Sliced child arrays.

Panics

Panics if data is not a [DataType::RunEndEncoded] array with a run_ends child of type R, or does not have exactly two children.

fn from(data: ArrayData) -> Self

Converts to this type from the input type.

impl<R: RunEndIndexType> From<RunArray<R>> for ArrayData

fn from(array: RunArray<R>) -> Self

Converts to this type from the input type.

impl<'a, T: RunEndIndexType> FromIterator<&'a str> for RunArray<T>

Constructs a RunArray from an iterator of strings.

Example:

use arrow_array::{RunArray, PrimitiveArray, StringArray, types::Int16Type};

let test = vec!["a", "a", "b", "c"];
let array: RunArray<Int16Type> = test.into_iter().collect();
assert_eq!(
    "RunArray {run_ends: [2, 3, 4], values: StringArray\n[\n  \"a\",\n  \"b\",\n  \"c\",\n]}\n",
    format!("{:?}", array)
);

fn from_iter<I: IntoIterator<Item = &'a str>>(iter: I) -> Self

Creates a value from an iterator.

impl<'a, T: RunEndIndexType> FromIterator<Option<&'a str>> for RunArray<T>

Constructs a RunArray from an iterator of optional strings.

Example:

use arrow_array::{RunArray, PrimitiveArray, StringArray, types::Int16Type};

let test = vec!["a", "a", "b", "c", "c"];
let array: RunArray<Int16Type> = test
    .iter()
    .map(|&x| if x == "b" { None } else { Some(x) })
    .collect();
assert_eq!(
    "RunArray {run_ends: [2, 3, 5], values: StringArray\n[\n  \"a\",\n  null,\n  \"c\",\n]}\n",
    format!("{:?}", array)
);

fn from_iter<I: IntoIterator<Item = Option<&'a str>>>(iter: I) -> Self

Creates a value from an iterator.

impl<R: RunEndIndexType> PartialEq for RunArray<R>

fn eq(&self, other: &Self) -> bool

Equality operator ==.

fn ne(&self, other: &Rhs) -> bool

Inequality operator !=.