Module reader 

Arrow IPC File and Stream Readers

Notes

The FileReader and StreamReader have similar interfaces, however the FileReader expects a reader that supports Seeking

Re-exports

use arrow_select::concat; 🔒

use flatbuffers::VectorIter; 🔒

use flatbuffers::VerifierOptions; 🔒

use std::collections::HashMap; 🔒

use std::collections::VecDeque; 🔒

use std::fmt; 🔒

use std::io::BufReader; 🔒

use std::io::Read; 🔒

use std::io::Seek; 🔒

use std::io::SeekFrom; 🔒

use std::sync::Arc; 🔒

use arrow_buffer::ArrowNativeType; 🔒

use arrow_buffer::BooleanBuffer; 🔒

use arrow_buffer::Buffer; 🔒

use arrow_buffer::MutableBuffer; 🔒

use arrow_buffer::NullBuffer; 🔒

use arrow_buffer::ScalarBuffer; 🔒

use arrow_data::ArrayData; 🔒

use arrow_data::ArrayDataBuilder; 🔒

use crate::compression::CompressionCodec; 🔒

use crate::compression::DecompressionContext; 🔒

use crate::gen::Message; 🔒

use crate::Block; 🔒

use crate::CONTINUATION_MARKER; 🔒

use crate::FieldNode; 🔒

use crate::MetadataVersion; 🔒

use arrow_array::*; 🔒

use arrow_schema::*; 🔒

use DataType::*; 🔒

Modules

stream 🔒

Structs

FileDecoder

A low-level, push-based interface for reading an IPC file

FileReader

Arrow File Reader

FileReaderBuilder

Build an Arrow FileReader with custom options.

MessageReader 🔒

A low-level construct that reads Message::Messages from a reader while re-using a buffer for metadata. This is composed into StreamReader.

RecordBatchDecoder

State for decoding Arrow arrays from an IPC RecordBatch structure to [RecordBatch]

StreamDecoder

A low-level interface for reading [RecordBatch] data from a stream of bytes

StreamReader

Arrow Stream Reader

Enums

DecoderState

IpcMessage 🔒

Representation of a fully parsed IpcMessage from the underlying stream. Parsing this kind of message is done by higher level constructs such as StreamReader, because fully interpreting the messages into a record batch or dictionary batch requires access to stream state such as schema and the full dictionary cache.

Constants

MAX_PREALLOC_BYTES 🔒

Upper bound on how much memory a single message length is allowed to reserve before any of the bytes it promises have been read.

Functions

check_variadic_counts_consumed 🔒

One variadic buffer count is consumed per BinaryView or Utf8View column in the schema, so any count left over means the message and the schema disagree.

child_field_with_dict_id 🔒

field_with_dict_id 🔒

first_field_with_dict_id 🔒

Returns the first field with dictionary id, in [Schema::fields_with_dict_id] order.

get_dictionary_values 🔒

Given a dictionary batch IPC message/body along with the full state of a stream including schema, dictionary cache, metadata, and other flags, this function will parse the buffer into an array of dictionary values.

parse_message 🔒

Parse an encapsulated message

read_block 🔒

Read the data for a given block

read_body_bounded 🔒

Reads exactly len bytes of message body, without reserving len before reading it.

read_buffer 🔒

Read a buffer based on offset and length From https://github.com/apache/arrow/blob/6a936c4ff5007045e86f65f1a6b6c3c955ad5103/format/Message.fbs#L58 Each constituent buffer is first compressed with the indicated compressor, and then written with the uncompressed length in the first 8 bytes as a 64-bit little-endian signed integer followed by the compressed buffer bytes (and then padding as required by the protocol). The uncompressed length may be set to -1 to indicate that the data that follows is not compressed, which can be useful for cases where compression does not yield appreciable savings.

read_dictionary

Read the dictionary from the buffer and provided metadata, updating the dictionaries_by_id with the resulting dictionary

read_dictionary_impl

Low-level version of read_dictionary with alignment and validation controls

read_footer_length

Read the footer length from the last 10 bytes of an Arrow IPC file

read_record_batch

Creates a record batch from binary data using the crate::RecordBatch indexes and the Schema.

reserve_declared

Reserves a message’s declared length before copying its first bytes into an empty buf, bounded as in read_body_bounded. Growing with each input buffer instead doubles the capacity, and the decoded batch, which is zero-copy over buf, keeps up to twice its size.

update_dictionaries 🔒

Updates the dictionaries_by_id with the provided dictionary values and id.