Arrays#

Building an array#

Available strategies#

As Arrow objects are immutable, they cannot be populated directly like for example a std::vector. Instead, several strategies can be used:

if the data already exists in memory with the right layout, you can wrap said memory inside arrow::Buffer instances and then construct a arrow::ArrowData describing the array;

See also

Memory Management
otherwise, the arrow::ArrayBuilder base class and its concrete subclasses help building up array data incrementally, without having to deal with details of the Arrow format yourself.

Using ArrayBuilder and its subclasses#

To build an Int64 Arrow array, we can use the arrow::Int64Builder class. In the following example, we build an array of the range 1 to 8 where the element that should hold the value 4 is nulled:

arrow::Int64Builder builder;
builder.Append(1);
builder.Append(2);
builder.Append(3);
builder.AppendNull();
builder.Append(5);
builder.Append(6);
builder.Append(7);
builder.Append(8);

auto maybe_array = builder.Finish();
if (!maybe_array.ok()) {
   // ... do something on array building failure
}
std::shared_ptr<arrow::Array> array = *maybe_array;

The resulting Array (which can be casted to the concrete arrow::Int64Array subclass if you want to access its values) then consists of two arrow::Buffers. The first buffer holds the null bitmap, which consists here of a single byte with the bits 1|1|1|1|0|1|1|1. As we use least-significant bit (LSB) numbering, this indicates that the fourth entry in the array is null. The second buffer is simply an int64_t array containing all the above values. As the fourth entry is null, the value at that position in the buffer is undefined.

Here is how you could access the concrete array’s contents:

// Cast the Array to its actual type to access its data
auto int64_array = std::static_pointer_cast<arrow::Int64Array>(array);

// Get the pointer to the null bitmap
const uint8_t* null_bitmap = int64_array->null_bitmap_data();

// Get the pointer to the actual data
const int64_t* data = int64_array->raw_values();

// Alternatively, given an array index, query its null bit and value directly
int64_t index = 2;
if (!int64_array->IsNull(index)) {
   int64_t value = int64_array->Value(index);
}

Note

arrow::Int64Array (respectively arrow::Int64Builder) is just a typedef, provided for convenience, of arrow::NumericArray<Int64Type> (respectively arrow::NumericBuilder<Int64Type>).

Performance#

While it is possible to build an array value-by-value as in the example above, to attain highest performance it is recommended to use the bulk appending methods (usually named AppendValues) in the concrete arrow::ArrayBuilder subclasses.

If you know the number of elements in advance, it is also recommended to presize the working area by calling the Resize() or Reserve() methods.

Here is how one could rewrite the above example to take advantage of those APIs:

arrow::Int64Builder builder;
// Make place for 8 values in total
builder.Reserve(8);
// Bulk append the given values (with a null in 4th place as indicated by the
// validity vector)
std::vector<bool> validity = {true, true, true, false, true, true, true, true};
std::vector<int64_t> values = {1, 2, 3, 0, 5, 6, 7, 8};
builder.AppendValues(values, validity);

auto maybe_array = builder.Finish();

If you still must append values one by one, some concrete builder subclasses have methods marked “Unsafe” that assume the working area has been correctly presized, and offer higher performance in exchange:

arrow::Int64Builder builder;
// Make place for 8 values in total
builder.Reserve(8);
builder.UnsafeAppend(1);
builder.UnsafeAppend(2);
builder.UnsafeAppend(3);
builder.UnsafeAppendNull();
builder.UnsafeAppend(5);
builder.UnsafeAppend(6);
builder.UnsafeAppend(7);
builder.UnsafeAppend(8);

auto maybe_array = builder.Finish();

Size Limitations and Recommendations#

Some array types are structurally limited to 32-bit sizes. This is the case for list arrays (which can hold up to 2^31 elements), string arrays and binary arrays (which can hold up to 2GB of binary data), at least. Some other array types can hold up to 2^63 elements in the C++ implementation, but other Arrow implementations can have a 32-bit size limitation for those array types as well.

For these reasons, it is recommended that huge data be chunked in subsets of more reasonable size.

Chunked Arrays#

A arrow::ChunkedArray is, like an array, a logical sequence of values; but unlike a simple array, a chunked array does not require the entire sequence to be physically contiguous in memory. Also, the constituents of a chunked array need not have the same size, but they must all have the same data type.

A chunked array is constructed by aggregating any number of arrays. Here we’ll build a chunked array with the same logical values as in the example above, but in two separate chunks:

std::vector<std::shared_ptr<arrow::Array>> chunks;
std::shared_ptr<arrow::Array> array;

// Build first chunk
arrow::Int64Builder builder;
builder.Append(1);
builder.Append(2);
builder.Append(3);
if (!builder.Finish(&array).ok()) {
   // ... do something on array building failure
}
chunks.push_back(std::move(array));

// Build second chunk
builder.Reset();
builder.AppendNull();
builder.Append(5);
builder.Append(6);
builder.Append(7);
builder.Append(8);
if (!builder.Finish(&array).ok()) {
   // ... do something on array building failure
}
chunks.push_back(std::move(array));

auto chunked_array = std::make_shared<arrow::ChunkedArray>(std::move(chunks));

assert(chunked_array->num_chunks() == 2);
// Logical length in number of values
assert(chunked_array->length() == 8);
assert(chunked_array->null_count() == 1);

Slicing#

Like for physical memory buffers, it is possible to make zero-copy slices of arrays and chunked arrays, to obtain an array or chunked array referring to some logical subsequence of the data. This is done by calling the arrow::Array::Slice() and arrow::ChunkedArray::Slice() methods, respectively.

Memory Management

Data Types