Module reader 

Arrow IPC File and Stream Readers

Notes

The FileReader and StreamReader have similar interfaces, however the FileReader expects a reader that supports Seeking

Modules

stream 🔒

Structs

FileDecoder

A low-level, push-based interface for reading an IPC file

FileReader

Arrow File Reader

FileReaderBuilder

Build an Arrow FileReader with custom options.

MessageReader 🔒

A low-level construct that reads Message::Messages from a reader while re-using a buffer for metadata. This is composed into StreamReader.

RecordBatchDecoder

State for decoding Arrow arrays from an IPC RecordBatch structure to [RecordBatch]

StreamDecoder

A low-level interface for reading [RecordBatch] data from a stream of bytes

StreamReader

Arrow Stream Reader

Enums

IpcMessage 🔒

Representation of a fully parsed IpcMessage from the underlying stream. Parsing this kind of message is done by higher level constructs such as StreamReader, because fully interpreting the messages into a record batch or dictionary batch requires access to stream state such as schema and the full dictionary cache.

Functions

get_dictionary_values 🔒

Given a dictionary batch IPC message/body along with the full state of a stream including schema, dictionary cache, metadata, and other flags, this function will parse the buffer into an array of dictionary values.

parse_message 🔒

Parse an encapsulated message

read_block 🔒

Read the data for a given block

read_buffer 🔒

Read a buffer based on offset and length From https://github.com/apache/arrow/blob/6a936c4ff5007045e86f65f1a6b6c3c955ad5103/format/Message.fbs#L58 Each constituent buffer is first compressed with the indicated compressor, and then written with the uncompressed length in the first 8 bytes as a 64-bit little-endian signed integer followed by the compressed buffer bytes (and then padding as required by the protocol). The uncompressed length may be set to -1 to indicate that the data that follows is not compressed, which can be useful for cases where compression does not yield appreciable savings.

read_dictionary

Read the dictionary from the buffer and provided metadata, updating the dictionaries_by_id with the resulting dictionary

read_dictionary_impl 🔒

read_footer_length

Read the footer length from the last 10 bytes of an Arrow IPC file

read_record_batch

Creates a record batch from binary data using the crate::RecordBatch indexes and the Schema.

update_dictionaries 🔒

Updates the dictionaries_by_id with the provided dictionary values and id.