Implementation Status¶
The following tables summarize the features available in the various official Arrow libraries. All libraries currently follow version 1.0.0 of the Arrow format. See Format Versioning and Stability for details about versioning. Unless otherwise stated, the Python, R, Ruby and C/GLib libraries follow the C++ Arrow library.
Data Types¶
Data type (primitive) |
C++ |
Java |
Go |
JavaScript |
C# |
Rust |
Julia |
|---|---|---|---|---|---|---|---|
Null |
✓ |
✓ |
✓ |
✓ |
✓ |
||
Boolean |
✓ |
✓ |
✓ |
✓ |
✓ |
✓ |
✓ |
Int8/16/32/64 |
✓ |
✓ |
✓ |
✓ |
✓ |
✓ |
✓ |
UInt8/16/32/64 |
✓ |
✓ |
✓ |
✓ |
✓ |
✓ |
✓ |
Float16 |
✓ |
✓ |
|||||
Float32/64 |
✓ |
✓ |
✓ |
✓ |
✓ |
✓ |
✓ |
Decimal128 |
✓ |
✓ |
✓ |
✓ |
✓ |
✓ |
|
Decimal256 |
✓ |
✓ |
✓ |
✓ |
|||
Date32/64 |
✓ |
✓ |
✓ |
✓ |
✓ |
✓ |
✓ |
Time32/64 |
✓ |
✓ |
✓ |
✓ |
✓ |
✓ |
✓ |
Timestamp |
✓ |
✓ |
✓ |
✓ |
✓ |
✓ |
✓ |
Duration |
✓ |
✓ |
✓ |
✓ |
✓ |
||
Interval |
✓ |
✓ |
✓ |
✓ |
✓ |
||
Fixed Size Binary |
✓ |
✓ |
✓ |
✓ |
✓ |
✓ |
✓ |
Binary |
✓ |
✓ |
✓ |
✓ |
✓ |
✓ |
✓ |
Large Binary |
✓ |
✓ |
✓ |
✓ |
✓ |
✓ |
|
Utf8 |
✓ |
✓ |
✓ |
✓ |
✓ |
✓ |
✓ |
Large Utf8 |
✓ |
✓ |
✓ |
✓ |
✓ |
✓ |
Data type (nested) |
C++ |
Java |
Go |
JavaScript |
C# |
Rust |
Julia |
|---|---|---|---|---|---|---|---|
Fixed Size List |
✓ |
✓ |
✓ |
✓ |
✓ |
✓ |
|
List |
✓ |
✓ |
✓ |
✓ |
✓ |
✓ |
✓ |
Large List |
✓ |
✓ |
✓ |
✓ |
|||
Struct |
✓ |
✓ |
✓ |
✓ |
✓ |
✓ |
✓ |
Map |
✓ |
✓ |
✓ |
✓ |
✓ |
||
Dense Union |
✓ |
✓ |
✓ |
||||
Sparse Union |
✓ |
✓ |
✓ |
Data type (special) |
C++ |
Java |
Go |
JavaScript |
C# |
Rust |
Julia |
|---|---|---|---|---|---|---|---|
Dictionary |
✓ |
✓ (1) |
✓ |
✓ (1) |
✓ (1) |
✓ (1) |
✓ |
Extension |
✓ |
✓ |
✓ |
✓ |
Notes:
(1) Nested dictionaries not supported
See also
The Arrow Columnar Format specification.
IPC Format¶
IPC Feature |
C++ |
Java |
Go |
JavaScript |
C# |
Rust |
Julia |
|---|---|---|---|---|---|---|---|
Arrow stream format |
✓ |
✓ |
✓ |
✓ |
✓ |
✓ |
✓ |
Arrow file format |
✓ |
✓ |
✓ |
✓ |
✓ |
✓ |
✓ |
Record batches |
✓ |
✓ |
✓ |
✓ |
✓ |
✓ |
✓ |
Dictionaries |
✓ |
✓ |
✓ |
✓ |
✓ |
✓ |
✓ |
Replacement dictionaries |
✓ |
✓ |
✓ |
✓ |
|||
Delta dictionaries |
✓ (1) |
✓ (1) |
✓ |
✓ |
|||
Tensors |
✓ |
||||||
Sparse tensors |
✓ |
||||||
Buffer compression |
✓ |
✓ (3) |
✓ |
✓ |
|||
Endianness conversion |
✓ (2) |
||||||
Custom schema metadata |
✓ |
✓ |
✓ |
✓ |
✓ |
✓ |
Notes:
(1) Delta dictionaries not supported on nested dictionaries
(2) Data with non-native endianness can be byte-swapped automatically when reading.
(3) LZ4 Codec currently is quite inefficient. ARROW-11901 tracks improving performance.
See also
The Serialization and Interprocess Communication (IPC) specification.
Flight RPC¶
Note
Flight RPC is still experimental.
Flight RPC Transport |
C++ |
Java |
Go |
JavaScript |
C# |
Rust |
Julia |
|---|---|---|---|---|---|---|---|
gRPC transport (grpc:, grpc+tcp:) |
✓ |
✓ |
✓ |
✓ |
|||
gRPC domain socket transport (grpc+unix:) |
✓ |
✓ |
✓ |
✓ |
|||
gRPC + TLS transport (grpc+tls:) |
✓ |
✓ |
✓ |
✓ |
|||
UCX transport (ucx:) |
✓ |
Supported features in the gRPC transport:
Flight RPC Feature |
C++ |
Java |
Go |
JavaScript |
C# |
Rust |
Julia |
|---|---|---|---|---|---|---|---|
All RPC methods |
✓ |
✓ |
✓ |
× (1) |
✓ |
||
Authentication handlers |
✓ |
✓ |
✓ |
✓ (2) |
✓ |
||
Call timeouts |
✓ |
✓ |
✓ |
✓ |
|||
Call cancellation |
✓ |
✓ |
✓ |
✓ |
|||
Concurrent client calls (3) |
✓ |
✓ |
✓ |
✓ |
✓ |
||
Custom middleware |
✓ |
✓ |
✓ |
✓ |
|||
RPC error codes |
✓ |
✓ |
✓ |
✓ |
✓ |
Supported features in the UCX transport:
Flight RPC Feature |
C++ |
Java |
Go |
JavaScript |
C# |
Rust |
Julia |
|---|---|---|---|---|---|---|---|
All RPC methods |
× (4) |
||||||
Authentication handlers |
|||||||
Call timeouts |
|||||||
Call cancellation |
|||||||
Concurrent client calls |
✓ (5) |
||||||
Custom middleware |
|||||||
RPC error codes |
✓ |
Notes:
(1) No support for handshake or DoExchange.
(2) Support using AspNetCore authentication handlers.
(3) Whether a single client can support multiple concurrent calls.
(4) Only support for DoExchange, DoGet, DoPut, and GetFlightInfo.
(5) Each concurrent call is a separate connection to the server (unlike gRPC where concurrent calls are multiplexed over a single connection). This will generally provide better throughput but consumes more resources both on the server and the client.
See also
The Arrow Flight RPC specification.
C Data Interface¶
Feature |
C++ |
Python |
R |
Rust |
Go |
Java |
C/GLib |
Ruby |
Julia |
|---|---|---|---|---|---|---|---|---|---|
Schema export |
✓ |
✓ |
✓ |
✓ |
✓ |
✓ |
✓ |
✓ |
|
Array export |
✓ |
✓ |
✓ |
✓ |
✓ |
✓ |
✓ |
✓ |
|
Schema import |
✓ |
✓ |
✓ |
✓ |
✓ |
✓ |
✓ |
✓ |
|
Array import |
✓ |
✓ |
✓ |
✓ |
✓ |
✓ |
✓ |
✓ |
See also
The C Data Interface specification.
C Stream Interface¶
Feature |
C++ |
Python |
R |
Rust |
Go |
Java |
C/GLib |
Ruby |
Julia |
|---|---|---|---|---|---|---|---|---|---|
Stream export |
✓ |
✓ |
✓ |
✓ |
✓ |
✓ |
|||
Stream import |
✓ |
✓ |
✓ |
✓ |
✓ |
✓ |
✓ |
See also
The C Stream Interface specification.
Third-Party Data Formats¶
Format |
C++ |
Java |
Go |
JavaScript |
C# |
Rust |
Julia |
|---|---|---|---|---|---|---|---|
Avro |
R |
||||||
CSV |
R/W |
R/W |
R/W |
R/W |
|||
ORC |
R/W |
R (2) |
|||||
Parquet |
R/W |
R (3) |
R/W |
R/W (1) |
Notes:
R = Read supported
W = Write supported
(1) Nested read/write not supported.
(2) Through JNI bindings. (Provided by
org.apache.arrow.orc:arrow-orc)(3) Through JNI bindings to Arrow C++ Datasets. (Provided by
org.apache.arrow:arrow-dataset)