High-Level Overview¶
The Arrow C++ library is comprised of different parts, each of which serves a specific purpose.
The physical layer¶
Memory management abstractions provide a uniform API over memory that may be allocated through various means, such as heap allocation, the memory mapping of a file or a static memory area. In particular, the buffer abstraction represents a contiguous area of physical data.
The one-dimensional layer¶
Data types govern the logical interpretation of physical data. Many operations in Arrow are parametered, at compile-time or at runtime, by a data type.
Arrays assemble one or several buffers with a data type, allowing to view them as a logical contiguous sequence of values (possibly nested).
Chunked arrays are a generalization of arrays, comprising several same-type arrays into a longer logical sequence of values.
The two-dimensional layer¶
Schemas describe a logical collection of several pieces of data, each with a distinct name and type, and optional metadata.
Tables are collections of chunked array in accordance to a schema. They are the most capable dataset-providing abstraction in Arrow.
Record batches are collections of contiguous arrays, described by a schema. They allow incremental construction or serialization of tables.
The compute layer¶
Datums are flexible dataset references, able to hold for example an array or table reference.
Kernels are specialized computation functions running in a loop over a given set of datums representing input and output parameters to the functions.
Acero (pronounced [aˈsɜɹo] / ah-SERR-oh) is a streaming execution engine that allows computation to be expressed as a graph of operators which can transform streams of data.
The IO layer¶
Streams allow untyped sequential or seekable access over external data of various kinds (for example compressed or memory-mapped).
The Inter-Process Communication (IPC) layer¶
A messaging format allows interchange of Arrow data between processes, using as few copies as possible.
The file formats layer¶
Reading and writing Arrow data from/to various file formats is possible, for example Parquet, CSV, Orc or the Arrow-specific Feather format.
The devices layer¶
Basic CUDA integration is provided, allowing to describe Arrow data backed by GPU-allocated memory.
The filesystem layer¶
A filesystem abstraction allows reading and writing data from different storage backends, such as the local filesystem or a S3 bucket.