Tabular Data#

See also

Table and RecordBatch API reference.

While arrays and chunked arrays represent a one-dimensional sequence of homogeneous values, data often comes in the form of two-dimensional sets of heterogeneous data (such as database tables, CSV files…). Arrow provides several abstractions to handle such data conveniently and efficiently.

Fields#

Fields are used to denote the particular columns of a table (and also the particular members of a nested data type such as arrow::StructType). A field, i.e. an instance of arrow::Field, holds together a data type, a field name and some optional metadata.

The recommended way to create a field is to call the arrow::field() factory function.

Schemas#

A schema describes the overall structure of a two-dimensional dataset such as a table. It holds a sequence of fields together with some optional schema-wide metadata (in addition to per-field metadata). The recommended way to create a schema is to call one the arrow::schema() factory function overloads:

// Create a schema describing datasets with two columns:
// a int32 column "A" and a utf8-encoded string column "B"
std::shared_ptr<arrow::Field> field_a, field_b;
std::shared_ptr<arrow::Schema> schema;

field_a = arrow::field("A", arrow::int32());
field_b = arrow::field("B", arrow::utf8());
schema = arrow::schema({field_a, field_b});

Tables#

A arrow::Table is a two-dimensional dataset with chunked arrays for columns, together with a schema providing field names. Also, each chunked column must have the same logical length in number of elements (although each column can be chunked in a different way).

Record Batches#

A arrow::RecordBatch is a two-dimensional dataset of a number of contiguous arrays, each the same length. Like a table, a record batch also has a schema which must match its arrays’ datatypes.

Record batches are a convenient unit of work for various serialization and computation functions, possibly incremental.

A graphical representation of an Arrow Table and a Record Batch, with structure as described in text above.

Record batches can be sent between implementations, such as via IPC or via the C Data Interface. Tables and chunked arrays, on the other hand, are concepts in the C++ implementation, not in the Arrow format itself, so they aren’t directly portable.

However, a table can be converted to and built from a sequence of record batches easily without needing to copy the underlying array buffers. A table can be streamed as an arbitrary number of record batches using a arrow::TableBatchReader. Conversely, a logical sequence of record batches can be assembled to form a table using one of the arrow::Table::FromRecordBatches() factory function overloads.