Apache Arrow nanoarrow 0.3.0 Release

Published 03 Oct 2023
By The Apache Arrow PMC (pmc)

The Apache Arrow team is pleased to announce the 0.3.0 release of Apache Arrow nanoarrow. This release covers 42 resolved issues from 4 contributors.

Release Highlights

See the Changelog for a detailed list of contributions to this release.

C library

The nanoarrow 0.3.0 release includes a number of bugfixes and improvements to the core C library and IPC C extension:

  • Improved bit packing/unpacking utilities and performance in tandem with an experiment to use bitmasks in pandas
  • Added support for building and consuming interval arrays
  • Fixed full validation of offset buffers for specific types of corrupted data that caused overflow
  • Fixed crashes that occurred for certain types of corrupted data and improved error messages that were misleading for other types of corrupted data.

R bindings

The nanoarrow R bindings are distributed as the nanoarrow package on CRAN. The 0.3.0 release of the R bindings includes improvements in type support and stability. Notably:

  • Conversion to/from bit64::integer64() is now supported
  • Warnings and errors for conversions that are invalid or may loose accuracy were improved
  • Extension types and extension type registration are now supported
  • Conversion of dictionary-encoded arrays to R vectors is now supported
  • Conversion of map arrays to R vectors is now supported
  • Improved performance of stream conversion to R vectors for streams of indeterminate length (e.g., database results from ADBC).

Experimental non-CPU support

With the addition of the Arrow C Device data interface comes an opportunity for nanoarrow to handle non-CPU (e.g., GPU) data. The nanoarrow_device extension was drafted in tandem with the specification and Arrow C++ implementation. While it is not an official part of the 0.3.0 release, it is available in an experimental state for those interested in its use and/or development.

Contributor experience

The 0.3.0 release features adoption of pre-commit hooks to improve consistency and contributor experience with a complex repository. Special thanks to @WillAyd for setting this up!

Contributors

This release consists of contributions from 4 contributors in addition to the invaluable advice and support of the Apache Arrow developer mailing list.

$ git shortlog -sn d4f038ce58655ba6e996cdae165f1b33c3919d51..apache-arrow-nanoarrow-0.3.0 | grep -v "GitHub Actions"
    40  Dewey Dunnington
     7  William Ayd
     2  Bryce Mecum
     1  Dane Pitkin