Apache Arrow 3.0.0 Release
25 Jan 2021
By The Apache Arrow PMC (pmc)
The Apache Arrow team is pleased to announce the 3.0.0 release. This covers over 3 months of development work and includes 666 resolved issues from 106 distinct contributors. See the Install Page to learn how to get the libraries for your platform.
The release notes below are not exhaustive and only expose selected highlights of the release. Many other bugfixes and improvements have been made: we refer you to the complete changelog.
Columnar Format Notes
The Decimal256 data type, which was already supported by the Arrow columnar format specification, is now implemented in C++ and Java (ARROW-9747).
Arrow Flight RPC notes
Authentication in C++/Java/Python has been overhauled, allowing more flexible authentication methods and use of standard headers. Support for cookies has also been added. The C++/Java implementations are now more permissive when parsing messages in order to interoperate better with other Flight implementations.
A basic Flight implementation for C#/.NET has been added. See the implementation status matrix for details.
The default memory pool can now be changed at runtime using the environment
ARROW_DEFAULT_MEMORY_POOL (ARROW-11009). The environment variable
is inspected at process startup. This is useful when trying to diagnose memory
consumption issues with Arrow.
STL-like iterators are now provided over concrete arrays. Those are useful for non-performance critical tasks, for example testing (ARROW-10776).
It is now possible to concatenate dictionary arrays with unequal dictionaries. The dictionaries are unified when concatenating, for supported data types (ARROW-5336).
Threads in a thread pool are now spawned lazily as needed for enqueued tasks, up to the configured capacity. They used to be spawned upfront on creation of the thread pool (ARROW-10038).
Comprehensive documentation for compute functions is now available: https://arrow.apache.org/docs/cpp/compute.html
Compute functions for string processing have been added for:
- splitting on whitespace (ASCII and Unicode flavors) and splitting on a pattern (ARROW-9991);
- trimming characters (ARROW-9128).
Behavior of the
is_in compute functions with nulls has been
changed for consistency (ARROW-10663).
Multiple-column sort kernels are now available for tables and record batches (ARROW-8199, ARROW-10796, ARROW-10790).
Performance of table filtering has been vastly improved (ARROW-10569).
Scalar arguments are now accepted for more compute functions.
quantile (ARROW-10831) and
is_nan (ARROW-11043) have been
added for numeric data.
any (ARROW-1846) and
all (ARROW-10301) have been
added for boolean data.
Expression hierarchy has simplified to a wrapper around literals, field references,
or calls to named functions. This enables usage of any compute function while filtering
with no boilerplate.
Parquet statistics are lazily parsed in
ParquetFileFragment for shorter construction time.
Conversion of string columns is now faster thanks to faster UTF-8 validation of small strings (ARROW-10313).
Conversion of floating-point columns is now faster thanks to optimized string-to-double conversion routines (ARROW-10328).
Parsing of ISO8601 timestamps is now more liberal: trailing zeros can be omitted in the fractional part (ARROW-10337).
Fixed a bug where null detection could give the wrong results on some platforms (ARROW-11067).
Added type inference for Date32 columns for values in the form
Fixed reading of compressed Feather files written with Arrow 0.17 (ARROW-11163).
S3 recursive tree walks now benefit from a parallel implementation, where reads of multiple child directories are now issued concurrently (ARROW-10788).
Improved empty directory detection to be mindful of differences between Amazon and Minio S3 implementations (ARROW-10942).
IPv6 host addresses are now supported (ARROW-10475).
It is now possible to emit dictionary deltas where possible using the IPC
stream writer. This is governed by a new variable in the
It is now possible to read wider tables, which used to fail due to reaching a limit during Flatbuffers verification (ARROW-10056).
Fixed reading of LZ4-compressed Parquet columns emitted by the Java Parquet implementation (ARROW-11301).
Fixed a bug where writing multiple batches of nullable nested strings to Parquet would not write any data in batches after the first one (ARROW-10493)
The Decimal256 data type can be read from and written to Parquet (ARROW-10607).
LargeString and LargeBinary data can now be written to Parquet (ARROW-10426).
The .NET package added initial support for Arrow Flight clients and servers. Support is enabled through two new NuGet packages Apache.Arrow.Flight (client) and Apache.Arrow.Flight.AspNetCore (server).
Also fixed an issue where ArrowStreamWriter wasn’t writing schema metadata to Arrow streams.
This is the first release to officially include an implementation for the Julia language. The pure Julia implementation includes support for wide coverage of the format specification. Additional details can be found in the julialang.org blog post.
Support for Python 3.9 was added (ARROW-10224), and support for Python 3.5 was removed (ARROW-5679).
Support for building manylinux1 packages has been removed (ARROW-11212). PyArrow continues to be available as manylinux2010 and manylinux2014 wheels.
The minimal required version for NumPy is now 1.16.6. Note that when upgrading NumPy to 1.20, you also need to upgrade pyarrow to 3.0.0 to ensure compatibility, as this pyarrow release fixed a compatibility issue with NumPy 1.20 (ARROW-10833).
Compute functions are now automatically exported from C++ to the
module, and they have docstrings matching their C++ definition.
iter_batches() method is now available for reading a Parquet file iteratively
Alternate memory pools (such as mimalloc, jemalloc or the C malloc-based memory pool) are now available from Python (ARROW-11049).
Fixed a potential deadlock when importing pandas from several threads (ARROW-10519).
See the C++ notes above for additional details.
This release contains new features for the Flight RPC wrapper, better
support for saving R metadata (including
sf spatial data) to Feather and
Parquet files, several significant improvements to speed and memory management,
and many other enhancements.
For more on what’s in the 3.0.0 R package, see the R changelog.
Ruby and C GLib notes
In Ruby binding, 256-bit decimal support and
Arrow::FixedBinaryArrayBuilder are added likewise C GLib below.
In the version 3.0.0 of C GLib consists of many new features.
A chunked array, a record batch, and a table support
sort_indices function as well as an array.
These functions including array’s support to specify sorting option.
garrow_array_sort_to_indices has been renamed to
garrow_array_sort_indices and the previous name has been deprecated.
GArrowField supports functions to handle metadata.
GArrowArrayBuilder supports to add single null, multiple nulls, single empty value, and multiple empty values.
GArrowFixedSizedBinaryArrayBuilder is newly supported.
256-bit decimal and extension types are newly supported. Filesystem module supports Mock, HDFS, S3 file systems. Dataset module supports CSV, IPC, and Parquet file formats.
Core Arrow Crate
The development of the arrow crate was focused on four main aspects:
- Make the crate usable in stable Rust
- Bug fixing and removal of
- Extend functionality to keep up with the specification
- Increase performance of existing kernels
Possibly the biggest news for this release is that all project crates, including arrow, parquet, and datafusion now build with stable Rust by default. Nightly / unstable Rust is still required when enabling the SIMD feature.
Parquet Arrow writer
The Parquet Writer for Arrow arrays is now available, allowing the Rust programs to easily read and write Parquet files and making it easier to integrate with the overall Arrow ecosystem. The reader and writer include both basic and nested type support (List, Dictionary, Struct)
First Class Arrow Flight IPC Support
This release the Arrow Flight IPC implementation in Rust became fully-featured enough to participate in the regular cross-language integration tests, thus ensuring Rust applications written using Arrow can interoperate with the rest of the ecosystem
There have been numerous performance improvements in this release across the board. This includes both kernel
operations, such as
cast, as well as more fundamental parts such as bitwise comparison
and reading and writing to CSV.
Increased Data Type Support
- Decimal data type for fixed-width decimal values
Improved operation support for nested structures Dictionary, and Lists (filter, take, etc)
- Added support for Date and time on FFI
- Added support for Binary type on FFI
- Added support for i64 sized arrays to “take” kernel
- Support for the i128 Decimal Type
- Added support to cast string to date
- Added API to create arrays out of existing arrays (e.g. for join, merge-sort, concatenate)
- The simd feature is now also available on aarch64
- BooleanArray is no longer a PrimitiveArray
- ArrowNativeType no longer includes bool since arrows boolean type is represented using bitpacking
- Several Buffer methods are now infallible instead of returning a Result
- DataType::List now contains a Field to track metadata about the contained elements
- PrimitiveArray::raw_values, values_slice and values methods got replaced by a values method returning a slice
- Buffer::data and raw_data were renamed to as_slice and as_ptr
- MutableBuffer::data_mut and freeze were renamed to as_slice_mut and into to be more consistent with the stdlib naming conventions
- The generic type parameter for BufferBuilder was changed from ArrowPrimitiveType to ArrowNativeType
In this release, we clarified that DataFusion will standardize on the PostgreSQL SQL dialect.
New SQL support:
- JOIN, LEFT JOIN, RIGHT JOIN
- COUNT DISTINCT
- CASE WHEN
- IS IN
- Nested SELECT statements
- Nested expressions in aggregations
- LOWER(), UPPER(), TRIM()
- SHA224(), SHA256(), SHA384(), SHA512()
There have been numerous performance improvements in this release:
- Optimizations for JOINs such as using vectorized hashing.
- We started with adding statistics and cost-based optimizations. We choose the smaller side of a join as the build side if possible.
- Improved parallelism when reading partitioned Parquet data sources
- Concurrent writes of CSV and Parquet partitions to file
The Parquet has the following improvements:
- Nested reading
- Support to write booleans
- Add support to write temporal types
Roadmap for 4.0.0
We have also started building up a shared community roadmap for 4.0: Apache Arrow: Crowd Sourced Rust Roadmap for Arrow 4.0, January 2021.