Apache Arrow nanoarrow 0.1 Release

Published 07 Mar 2023
By The Apache Arrow PMC (pmc)

The Apache Arrow team is pleased to announce the 0.1.0 release of Apache Arrow nanoarrow. This initial release covers 31 resolved issues from 6 contributors.

Release Highlights

This initial release includes the following:

A C library bundled as two files (nanoarrow.c and nanoarrow.h).
An R package providing bindings for users of the R programming language.

See the Changelog for a detailed list of contributions leading up to the initial release.

Why nanoarrow?

The adoption of the Arrow C Data Interface and the Arrow C Stream Interface since their introduction have been impressive and enthusiastic: not only have Arrow language bindings adopted the standard to pass data among themselves, a growing number of high-profile libraries like GDAL and DuckDB use the standard to improve performance and provide an ABI-stable interface to tabular input and output.

GDAL and DuckDB are fortunate to have hard-working and forward-thinking maintainers that were motivated to provide support for the Arrow C Data and Stream interfaces even though the code to do so required an intimate knowledge of both the interface and the columnar specification on which it is based.

The vision of nanoarrow is that it should be trivial for a library or application to implement an Arrow-based interface: if a library consumes or produces tabular data, Arrow should be the first place developers look. Developers shouldn’t have to be familiar with the details of the columnar specification—nor should they have to take on any build-time dependencies—to get started.

The Arrow Database Connectivity (ADBC) specification is a good example of such a project, and provided a strong motivator for the development of nanoarrow: at the heart of ADBC is the idea of a core “driver manager” and database-specific drivers that are distributed as independent C/C++/Python/R/Java/Go projects. At least in R and Python, embedding an existing Arrow implementation (e.g., Arrow C++) is challenging in the context of multiple packages intended to be loaded into the same process. As of this writing, ADBC includes nanoarrow-based SQLite and PostgreSQL drivers and a nanoarrow-based validation suite for drivers.

Using nanoarrow in C

The nanoarrow C library is distributed as two files (nanoarrow.h and nanoarrow.c) that can be copied and vendored into an existing code base. This results in a static library of about 50 KB and builds in less than a second. Some features that nanoarrow provides are:

Helpers to create types, schemas, and metadata
Growable buffers, including the option for custom allocators/deallocators.
Bitmap (i.e., bitpacked boolean) utilities
An API for building arrays from buffers
An API for building arrays element-wise
An API to extract elements element-wise from an existing array.

For example, one can build an integer array element-wise:

#include "nanoarrow.h"

int make_simple_array(struct ArrowArray* array_out, struct ArrowSchema* schema_out) {
  struct ArrowError error;
  array_out->release = NULL;
  schema_out->release = NULL;

  NANOARROW_RETURN_NOT_OK(ArrowArrayInitFromType(array_out, NANOARROW_TYPE_INT32));

  NANOARROW_RETURN_NOT_OK(ArrowArrayStartAppending(array_out));
  NANOARROW_RETURN_NOT_OK(ArrowArrayAppendInt(array_out, 1));
  NANOARROW_RETURN_NOT_OK(ArrowArrayAppendInt(array_out, 2));
  NANOARROW_RETURN_NOT_OK(ArrowArrayAppendInt(array_out, 3));
  NANOARROW_RETURN_NOT_OK(ArrowArrayFinishBuilding(array_out, &error));

  NANOARROW_RETURN_NOT_OK(ArrowSchemaInitFromType(schema_out, NANOARROW_TYPE_INT32));

  return NANOARROW_OK;
}

Similarly, one can extract elements from an array:

#include <stdio.h>
#include "nanoarrow.h"

int print_simple_array(struct ArrowArray* array, struct ArrowSchema* schema) {
  struct ArrowError error;
  struct ArrowArrayView array_view;
  NANOARROW_RETURN_NOT_OK(ArrowArrayViewInitFromSchema(&array_view, schema, &error));

  if (array_view.storage_type != NANOARROW_TYPE_INT32) {
    printf("Array has storage that is not int32\n");
  }

  int result = ArrowArrayViewSetArray(&array_view, array, &error);
  if (result != NANOARROW_OK) {
    ArrowArrayViewReset(&array_view);
    return result;
  }

  for (int64_t i = 0; i < array->length; i++) {
    printf("%d\n", (int)ArrowArrayViewGetIntUnsafe(&array_view, i));
  }

  ArrowArrayViewReset(&array_view);
  return NANOARROW_OK;
}

Using nanoarrow in C++, R, and Python

Recognizing that many projects for which nanoarrow may be useful will have access a higher-level runtime than C, there are experiments to provide these users with a minimal set of useful tools.

For C++ projects, an experimental “nanoarrow.hpp” interface provides unique_ptr-like wrappers for nanoarrow C objects to reduce the verbosity of using the nanoarrow API. For example, the previous print_simple_array() implementation would collapse to:

#include <stdio.h>
#include "nanoarrow.hpp"

int print_simple_array2(struct ArrowArray* array, struct ArrowSchema* schema) {
  struct ArrowError error;
  nanoarrow::UniqueArrayView array_view;
  NANOARROW_RETURN_NOT_OK(ArrowArrayViewInitFromSchema(array_view.get(), schema, &error));
  NANOARROW_RETURN_NOT_OK(ArrowArrayViewSetArray(array_view.get(), array, &error));
  for (int64_t i = 0; i < array->length; i++) {
    printf("%d\n", (int)ArrowArrayViewGetIntUnsafe(array_view.get(), i));
  }
  return NANOARROW_OK;
}

For R packages, experimental R bindings provide a limited set of conversions between R vectors and Arrow arrays such that R bindings for a library with an Arrow-based interface do not need to provide this behaviour themselves. Additional features include printing and validating the content of the C structures at the heart of the C Data and C Stream interfaces to facilitate the development of bindings to Arrow-based libraries.

# install.packages("remotes")
remotes::install_github("apache/arrow-nanoarrow/r", build = FALSE)
library(nanoarrow)

as_nanoarrow_array(1:5)
#> <nanoarrow_array int32[5]>
#>  $ length    : int 5
#>  $ null_count: int 0
#>  $ offset    : int 0
#>  $ buffers   :List of 2
#>   ..$ :<nanoarrow_buffer_validity[0 b] at 0x0>
#>   ..$ :<nanoarrow_buffer_data_int32[20 b] at 0x135d13c28>
#>  $ dictionary: NULL
#>  $ children  : list()

A Python package skeleton exists in the nanoarrow repository and further functionality may be added once the C library interface has stabilized.

Try nanoarrow

For any interested in giving nanoarrow a try, the easiest way to get started is to clone the nanoarrow repository from GitHub and build/modify the minimal CMake build example. For applied usage, one can refer to the ADBC SQLite driver and the ADBC PostgreSQL driver.

Contributors

This initial release consists of contributions from 6 contributors in addition to the invaluable advice and support of the Apache Arrow developer mailing list.

$ git shortlog -sn 8339114637919b661c1c8fae6764ceed532c935e..apache-arrow-nanoarrow-0.1.0 | grep -v "GitHub Actions"
Dewey Dunnington
David Li
Dirk Eddelbuettel
Dane Pitkin
Jonathan Keane
Joris Van den Bossche