Basic Arrow Data Structures

Apache Arrow provides fundamental data structures for representing data: Array, ChunkedArray, RecordBatch, and Table. This article shows how to construct these data structures from primitive data types; specifically, we will work with integers of varying size representing days, months, and years. We will use them to create the following data structures:

  1. Arrow Arrays

  2. ChunkedArrays

  3. RecordBatch, from Arrays

  4. Table, from ChunkedArrays

Pre-requisites

Before continuing, make sure you have:

  1. An Arrow installation, which you can set up here: Using Arrow C++ in your own project

  2. Understanding of how to use basic C++ data structures

  3. Understanding of basic C++ data types

Setup

Before trying out Arrow, we need to fill in a couple gaps:

  1. We need to include necessary headers.

  2. A main() is needed to glue things together.

Includes

First, as ever, we need some includes. We’ll get iostream for output, then import Arrow’s basic functionality from api.h, like so:

#include <arrow/api.h>

#include <iostream>

Main()

Next, we need a main() – a common pattern with Arrow looks like the following:

int main() {
  arrow::Status st = RunMain();
  if (!st.ok()) {
    std::cerr << st << std::endl;
    return 1;
  }
  return 0;
}

This allows us to easily use Arrow’s error-handling macros, which will return back to main() with a arrow::Status object if a failure occurs – and this main() will report the error. Note that this means Arrow never raises exceptions, instead relying upon returning Status. For more on that, read here: Conventions.

To accompany this main(), we have a RunMain() from which any Status objects can return – this is where we’ll write the rest of the program:

arrow::Status RunMain() {

Making an Arrow Array

Building int8 Arrays

Given that we have some data in standard C++ arrays, and want to use Arrow, we need to move the data from said arrays into Arrow arrays. We still guarantee contiguity of memory in an Array, so no worries about a performance loss when using Array vs C++ arrays. The easiest way to construct an Array uses an ArrayBuilder.

See also

Arrays for more technical details on Array

The following code initializes an ArrayBuilder for an Array that will hold 8 bit integers. Specifically, it uses the AppendValues() method, present in concrete arrow::ArrayBuilder subclasses, to fill the ArrayBuilder with the contents of a standard C++ array. Note the use of ARROW_RETURN_NOT_OK. If AppendValues() fails, this macro will return to main(), which will print out the meaning of the failure.

  // Builders are the main way to create Arrays in Arrow from existing values that are not
  // on-disk. In this case, we'll make a simple array, and feed that in.
  // Data types are important as ever, and there is a Builder for each compatible type;
  // in this case, int8.
  arrow::Int8Builder int8builder;
  int8_t days_raw[5] = {1, 12, 17, 23, 28};
  // AppendValues, as called, puts 5 values from days_raw into our Builder object.
  ARROW_RETURN_NOT_OK(int8builder.AppendValues(days_raw, 5));

Given an ArrayBuilder has the values we want in our Array, we can use ArrayBuilder::Finish() to output the final structure to an Array – specifically, we output to a std::shared_ptr<arrow::Array>. Note the use of ARROW_ASSIGN_OR_RAISE in the following code. Finish() outputs a arrow::Result object, which ARROW_ASSIGN_OR_RAISE can process. If the method fails, it will return to main() with a Status that will explain what went wrong. If it succeeds, then it will assign the final output to the left-hand variable.

  // We only have a Builder though, not an Array -- the following code pushes out the
  // built up data into a proper Array.
  std::shared_ptr<arrow::Array> days;
  ARROW_ASSIGN_OR_RAISE(days, int8builder.Finish());

As soon as ArrayBuilder has had its Finish method called, its state resets, so it can be used again, as if it was fresh. Thus, we repeat the process above for our second array:

  // Builders clear their state every time they fill an Array, so if the type is the same,
  // we can re-use the builder. We do that here for month values.
  int8_t months_raw[5] = {1, 3, 5, 7, 1};
  ARROW_RETURN_NOT_OK(int8builder.AppendValues(months_raw, 5));
  std::shared_ptr<arrow::Array> months;
  ARROW_ASSIGN_OR_RAISE(months, int8builder.Finish());

Building int16 Arrays

An ArrayBuilder has its type specified at the time of declaration. Once this is done, it cannot have its type changed. We have to make a new one when we switch to year data, which requires a 16-bit integer at the minimum. Of course, there’s an ArrayBuilder for that. It uses the exact same methods, but with the new data type:

  // Now that we change to int16, we use the Builder for that data type instead.
  arrow::Int16Builder int16builder;
  int16_t years_raw[5] = {1990, 2000, 1995, 2000, 1995};
  ARROW_RETURN_NOT_OK(int16builder.AppendValues(years_raw, 5));
  std::shared_ptr<arrow::Array> years;
  ARROW_ASSIGN_OR_RAISE(years, int16builder.Finish());

Now, we have three Arrow Arrays, with some variance in type.

Making a RecordBatch

A columnar data format only really comes into play when you have a table. So, let’s make one. The first kind we’ll make is the RecordBatch – this uses Arrays internally, which means all data will be contiguous within each column, but any appending or concatenating will require copying. Making a RecordBatch has two steps, given existing Arrays:

  1. Defining a Schema

  2. Loading the Schema and Arrays into the constructor

Defining a Schema

To get started making a RecordBatch, we first need to define characteristics of the columns, each represented by a Field instance. Each Field contains a name and datatype for its associated column; then, a Schema groups them together and sets the order of the columns, like so:

  // Now, we want a RecordBatch, which has columns and labels for said columns.
  // This gets us to the 2d data structures we want in Arrow.
  // These are defined by schema, which have fields -- here we get both those object types
  // ready.
  std::shared_ptr<arrow::Field> field_day, field_month, field_year;
  std::shared_ptr<arrow::Schema> schema;

  // Every field needs its name and data type.
  field_day = arrow::field("Day", arrow::int8());
  field_month = arrow::field("Month", arrow::int8());
  field_year = arrow::field("Year", arrow::int16());

  // The schema can be built from a vector of fields, and we do so here.
  schema = arrow::schema({field_day, field_month, field_year});

Building a RecordBatch

With data in Arrays from the previous section, and column descriptions in our Schema from the previous step, we can make the RecordBatch. Note that the length of the columns is necessary, and the length is shared by all columns.

  // With the schema and Arrays full of data, we can make our RecordBatch! Here,
  // each column is internally contiguous. This is in opposition to Tables, which we'll
  // see next.
  std::shared_ptr<arrow::RecordBatch> rbatch;
  // The RecordBatch needs the schema, length for columns, which all must match,
  // and the actual data itself.
  rbatch = arrow::RecordBatch::Make(schema, days->length(), {days, months, years});

  std::cout << rbatch->ToString();

Now, we have our data in a nice tabular form, safely within the RecordBatch. What we can do with this will be discussed in the later tutorials.

Making a ChunkedArray

Let’s say that we want an array made up of sub-arrays, because it can be useful for avoiding data copies when concatenating, for parallelizing work, for fitting each chunk cutely into cache, or for exceeding the 2,147,483,647 row limit in a standard Arrow Array. For this, Arrow offers ChunkedArray, which can be made up of individual Arrow Arrays. In this example, we can reuse the arrays we made earlier in part of our chunked array, allowing us to extend them without having to copy data. So, let’s build a few more Arrays, using the same builders for ease of use:

  // Now, let's get some new arrays! It'll be the same datatypes as above, so we re-use
  // Builders.
  int8_t days_raw2[5] = {6, 12, 3, 30, 22};
  ARROW_RETURN_NOT_OK(int8builder.AppendValues(days_raw2, 5));
  std::shared_ptr<arrow::Array> days2;
  ARROW_ASSIGN_OR_RAISE(days2, int8builder.Finish());

  int8_t months_raw2[5] = {5, 4, 11, 3, 2};
  ARROW_RETURN_NOT_OK(int8builder.AppendValues(months_raw2, 5));
  std::shared_ptr<arrow::Array> months2;
  ARROW_ASSIGN_OR_RAISE(months2, int8builder.Finish());

  int16_t years_raw2[5] = {1980, 2001, 1915, 2020, 1996};
  ARROW_RETURN_NOT_OK(int16builder.AppendValues(years_raw2, 5));
  std::shared_ptr<arrow::Array> years2;
  ARROW_ASSIGN_OR_RAISE(years2, int16builder.Finish());

In order to support an arbitrary amount of Arrays in the construction of the ChunkedArray, Arrow supplies ArrayVector. This provides a vector for Arrays, and we’ll use it here to prepare to make a ChunkedArray:

  // ChunkedArrays let us have a list of arrays, which aren't contiguous
  // with each other. First, we get a vector of arrays.
  arrow::ArrayVector day_vecs{days, days2};

In order to leverage Arrow, we do need to take that last step, and move into a ChunkedArray:

  // Then, we use that to initialize a ChunkedArray, which can be used with other
  // functions in Arrow! This is good, since having a normal vector of arrays wouldn't
  // get us far.
  std::shared_ptr<arrow::ChunkedArray> day_chunks =
      std::make_shared<arrow::ChunkedArray>(day_vecs);

With a ChunkedArray for our day values, we now just need to repeat the process for the month and year data:

  // Repeat for months.
  arrow::ArrayVector month_vecs{months, months2};
  std::shared_ptr<arrow::ChunkedArray> month_chunks =
      std::make_shared<arrow::ChunkedArray>(month_vecs);

  // Repeat for years.
  arrow::ArrayVector year_vecs{years, years2};
  std::shared_ptr<arrow::ChunkedArray> year_chunks =
      std::make_shared<arrow::ChunkedArray>(year_vecs);

With that, we are left with three ChunkedArrays, varying in type.

Making a Table

One particularly useful thing we can do with the ChunkedArrays from the previous section is creating Tables. Much like a RecordBatch, a Table stores tabular data. However, a Table does not guarantee contiguity, due to being made up of ChunkedArrays. This can be useful for logic, paralellizing work, for fitting chunks into cache, or exceeding the 2,147,483,647 row limit present in Array and, thus, RecordBatch.

If you read up to RecordBatch, you may note that the Table constructor in the following code is effectively identical, it just happens to put the length of the columns in position 3, and makes a Table. We re-use the Schema from before, and make our Table:

  // A Table is the structure we need for these non-contiguous columns, and keeps them
  // all in one place for us so we can use them as if they were normal arrays.
  std::shared_ptr<arrow::Table> table;
  table = arrow::Table::Make(schema, {day_chunks, month_chunks, year_chunks}, 10);

  std::cout << table->ToString();

Now, we have our data in a nice tabular form, safely within the Table. What we can do with this will be discussed in the later tutorials.

Ending Program

At the end, we just return Status::OK(), so the main() knows that we’re done, and that everything’s okay.

  return arrow::Status::OK();
}

Wrapping Up

With that, you’ve created the fundamental data structures in Arrow, and can proceed to getting them in and out of a program with file I/O in the next article.

Refer to the below for a copy of the complete code:

 19// (Doc section: Includes)
 20#include <arrow/api.h>
 21
 22#include <iostream>
 23// (Doc section: Includes)
 24
 25// (Doc section: RunMain Start)
 26arrow::Status RunMain() {
 27  // (Doc section: RunMain Start)
 28  // (Doc section: int8builder 1 Append)
 29  // Builders are the main way to create Arrays in Arrow from existing values that are not
 30  // on-disk. In this case, we'll make a simple array, and feed that in.
 31  // Data types are important as ever, and there is a Builder for each compatible type;
 32  // in this case, int8.
 33  arrow::Int8Builder int8builder;
 34  int8_t days_raw[5] = {1, 12, 17, 23, 28};
 35  // AppendValues, as called, puts 5 values from days_raw into our Builder object.
 36  ARROW_RETURN_NOT_OK(int8builder.AppendValues(days_raw, 5));
 37  // (Doc section: int8builder 1 Append)
 38
 39  // (Doc section: int8builder 1 Finish)
 40  // We only have a Builder though, not an Array -- the following code pushes out the
 41  // built up data into a proper Array.
 42  std::shared_ptr<arrow::Array> days;
 43  ARROW_ASSIGN_OR_RAISE(days, int8builder.Finish());
 44  // (Doc section: int8builder 1 Finish)
 45
 46  // (Doc section: int8builder 2)
 47  // Builders clear their state every time they fill an Array, so if the type is the same,
 48  // we can re-use the builder. We do that here for month values.
 49  int8_t months_raw[5] = {1, 3, 5, 7, 1};
 50  ARROW_RETURN_NOT_OK(int8builder.AppendValues(months_raw, 5));
 51  std::shared_ptr<arrow::Array> months;
 52  ARROW_ASSIGN_OR_RAISE(months, int8builder.Finish());
 53  // (Doc section: int8builder 2)
 54
 55  // (Doc section: int16builder)
 56  // Now that we change to int16, we use the Builder for that data type instead.
 57  arrow::Int16Builder int16builder;
 58  int16_t years_raw[5] = {1990, 2000, 1995, 2000, 1995};
 59  ARROW_RETURN_NOT_OK(int16builder.AppendValues(years_raw, 5));
 60  std::shared_ptr<arrow::Array> years;
 61  ARROW_ASSIGN_OR_RAISE(years, int16builder.Finish());
 62  // (Doc section: int16builder)
 63
 64  // (Doc section: Schema)
 65  // Now, we want a RecordBatch, which has columns and labels for said columns.
 66  // This gets us to the 2d data structures we want in Arrow.
 67  // These are defined by schema, which have fields -- here we get both those object types
 68  // ready.
 69  std::shared_ptr<arrow::Field> field_day, field_month, field_year;
 70  std::shared_ptr<arrow::Schema> schema;
 71
 72  // Every field needs its name and data type.
 73  field_day = arrow::field("Day", arrow::int8());
 74  field_month = arrow::field("Month", arrow::int8());
 75  field_year = arrow::field("Year", arrow::int16());
 76
 77  // The schema can be built from a vector of fields, and we do so here.
 78  schema = arrow::schema({field_day, field_month, field_year});
 79  // (Doc section: Schema)
 80
 81  // (Doc section: RBatch)
 82  // With the schema and Arrays full of data, we can make our RecordBatch! Here,
 83  // each column is internally contiguous. This is in opposition to Tables, which we'll
 84  // see next.
 85  std::shared_ptr<arrow::RecordBatch> rbatch;
 86  // The RecordBatch needs the schema, length for columns, which all must match,
 87  // and the actual data itself.
 88  rbatch = arrow::RecordBatch::Make(schema, days->length(), {days, months, years});
 89
 90  std::cout << rbatch->ToString();
 91  // (Doc section: RBatch)
 92
 93  // (Doc section: More Arrays)
 94  // Now, let's get some new arrays! It'll be the same datatypes as above, so we re-use
 95  // Builders.
 96  int8_t days_raw2[5] = {6, 12, 3, 30, 22};
 97  ARROW_RETURN_NOT_OK(int8builder.AppendValues(days_raw2, 5));
 98  std::shared_ptr<arrow::Array> days2;
 99  ARROW_ASSIGN_OR_RAISE(days2, int8builder.Finish());
100
101  int8_t months_raw2[5] = {5, 4, 11, 3, 2};
102  ARROW_RETURN_NOT_OK(int8builder.AppendValues(months_raw2, 5));
103  std::shared_ptr<arrow::Array> months2;
104  ARROW_ASSIGN_OR_RAISE(months2, int8builder.Finish());
105
106  int16_t years_raw2[5] = {1980, 2001, 1915, 2020, 1996};
107  ARROW_RETURN_NOT_OK(int16builder.AppendValues(years_raw2, 5));
108  std::shared_ptr<arrow::Array> years2;
109  ARROW_ASSIGN_OR_RAISE(years2, int16builder.Finish());
110  // (Doc section: More Arrays)
111
112  // (Doc section: ArrayVector)
113  // ChunkedArrays let us have a list of arrays, which aren't contiguous
114  // with each other. First, we get a vector of arrays.
115  arrow::ArrayVector day_vecs{days, days2};
116  // (Doc section: ArrayVector)
117  // (Doc section: ChunkedArray Day)
118  // Then, we use that to initialize a ChunkedArray, which can be used with other
119  // functions in Arrow! This is good, since having a normal vector of arrays wouldn't
120  // get us far.
121  std::shared_ptr<arrow::ChunkedArray> day_chunks =
122      std::make_shared<arrow::ChunkedArray>(day_vecs);
123  // (Doc section: ChunkedArray Day)
124
125  // (Doc section: ChunkedArray Month Year)
126  // Repeat for months.
127  arrow::ArrayVector month_vecs{months, months2};
128  std::shared_ptr<arrow::ChunkedArray> month_chunks =
129      std::make_shared<arrow::ChunkedArray>(month_vecs);
130
131  // Repeat for years.
132  arrow::ArrayVector year_vecs{years, years2};
133  std::shared_ptr<arrow::ChunkedArray> year_chunks =
134      std::make_shared<arrow::ChunkedArray>(year_vecs);
135  // (Doc section: ChunkedArray Month Year)
136
137  // (Doc section: Table)
138  // A Table is the structure we need for these non-contiguous columns, and keeps them
139  // all in one place for us so we can use them as if they were normal arrays.
140  std::shared_ptr<arrow::Table> table;
141  table = arrow::Table::Make(schema, {day_chunks, month_chunks, year_chunks}, 10);
142
143  std::cout << table->ToString();
144  // (Doc section: Table)
145
146  // (Doc section: Ret)
147  return arrow::Status::OK();
148}
149// (Doc section: Ret)
150
151// (Doc section: Main)
152int main() {
153  arrow::Status st = RunMain();
154  if (!st.ok()) {
155    std::cerr << st << std::endl;
156    return 1;
157  }
158  return 0;
159}
160
161// (Doc section: Main)

previous

Conventions

next

Arrow File I/O