Input / output and filesystems#

Arrow provides a range of C++ interfaces abstracting the concrete details of input / output operations. They operate on streams of untyped binary data. Those abstractions are used for various purposes such as reading CSV or Parquet data, transmitting IPC streams, and more.

See also

API reference for input/output facilities.

Reading binary data#

Interfaces for reading binary data come in two flavours:

  • Sequential reading: the InputStream interface provides Read methods; it is recommended to Read to a Buffer as it may in some cases avoid a memory copy.

  • Random access reading: the RandomAccessFile interface provides additional facilities for positioning and, most importantly, the ReadAt methods which allow parallel reading from multiple threads.

Concrete implementations are available for in-memory reads, unbuffered file reads, memory-mapped file reads, buffered reads, compressed reads.

Writing binary data#

Writing binary data is mostly done through the OutputStream interface.

Concrete implementations are available for in-memory writes, unbuffered file writes, memory-mapped file writes, buffered writes, compressed writes.

Filesystems#

The filesystem interface allows abstracted access over various data storage backends such as the local filesystem or a S3 bucket. It provides input and output streams as well as directory operations.

The filesystem interface exposes a simplified view of the underlying data storage. Data paths are represented as abstract paths, which are /-separated, even on Windows, and shouldn’t include special path components such as . and ... Symbolic links, if supported by the underlying storage, are automatically dereferenced. Only basic metadata about file entries, such as the file size and modification time, is made available.

Concrete implementations are available for local filesystem access, HDFS, Amazon S3-compatible storage and Google Cloud Storage.

Note

Tasks that use filesystems will typically run on the I/O thread pool. For filesystems that support high levels of concurrency you may get a benefit from increasing the size of the I/O thread pool.