Apache Arrow nanoarrow 0.3.0 Release
03 Oct 2023
By The Apache Arrow PMC (pmc)
The Apache Arrow team is pleased to announce the 0.3.0 release of Apache Arrow nanoarrow. This release covers 42 resolved issues from 4 contributors.
See the Changelog for a detailed list of contributions to this release.
The nanoarrow 0.3.0 release includes a number of bugfixes and improvements to the core C library and IPC C extension:
- Improved bit packing/unpacking utilities and performance in tandem with an experiment to use bitmasks in pandas
- Added support for building and consuming interval arrays
- Fixed full validation of offset buffers for specific types of corrupted data that caused overflow
- Fixed crashes that occurred for certain types of corrupted data and improved error messages that were misleading for other types of corrupted data.
The nanoarrow R bindings are distributed as the
nanoarrow package on
CRAN. The 0.3.0 release of the R bindings includes
improvements in type support and stability. Notably:
- Conversion to/from
bit64::integer64()is now supported
- Warnings and errors for conversions that are invalid or may loose accuracy were improved
- Extension types and extension type registration are now supported
- Conversion of dictionary-encoded arrays to R vectors is now supported
- Conversion of map arrays to R vectors is now supported
- Improved performance of stream conversion to R vectors for streams of indeterminate length (e.g., database results from ADBC).
Experimental non-CPU support
With the addition of the Arrow C Device data interface comes an opportunity for nanoarrow to handle non-CPU (e.g., GPU) data. The nanoarrow_device extension was drafted in tandem with the specification and Arrow C++ implementation. While it is not an official part of the 0.3.0 release, it is available in an experimental state for those interested in its use and/or development.
The 0.3.0 release features adoption of
pre-commit hooks to improve consistency
and contributor experience with a complex repository. Special thanks to
@WillAyd for setting this up!
This release consists of contributions from 4 contributors in addition to the invaluable advice and support of the Apache Arrow developer mailing list.
$ git shortlog -sn d4f038ce58655ba6e996cdae165f1b33c3919d51..apache-arrow-nanoarrow-0.3.0 | grep -v "GitHub Actions" 40 Dewey Dunnington 7 William Ayd 2 Bryce Mecum 1 Dane Pitkin