Module writer 

Arrow IPC File and Stream Writers

Notes

FileWriter and StreamWriter have similar interfaces, however the FileWriter expects a reader that supports Seeking

Structs

CompressionContext

Additional context that may be needed for compression.

DictionaryTracker

Keeps track of dictionaries that have been written, to avoid emitting the same dictionary multiple times.

EncodedData

Stores the encoded data, which is an crate::Message, and optional Arrow data

FileWriter

Arrow File Writer

IpcDataGenerator

Handles low level details of encoding [Array] and [Schema] into the Arrow IPC Format.

IpcMetadataBuilder 🔒

Accumulates the IPC metadata produced by write_array_data.

IpcWriteMetadata 🔒

Per-message sizes produced by IpcDataGenerator::write.

IpcWriteOptions

IPC write options used to control the behaviour of the IpcDataGenerator

StreamWriter

Arrow Stream Writer

Enums

DictionaryComparison 🔒

Describes how two dictionary arrays compare to each other.

DictionaryHandling

Controls how dictionaries are handled in Arrow IPC messages

DictionaryUpdate

Describes what kind of update took place after a call to DictionaryTracker::insert.

EncodedBuffer 🔒

A single buffer segment ready to be written to the output stream.

IpcBodySink 🔒

Destination for the raw Arrow data bytes (the IPC message body) produced by write_array_data.

Constants

PADDING 🔒

Functions

append_variadic_buffer_counts 🔒

buffer_need_truncate 🔒

Whether to truncate the buffer

compare_dictionaries 🔒

encode_sink_buffer 🔒

Encodes a single Arrow [Buffer] into the IPC body and records its metadata.

estimate_encoded_buffer_count 🔒

Estimates the number of EncodedBuffer segments that write_array_data will produce for a column of the given type.

get_buffer_element_width 🔒

Returns byte width for a buffer spec. Only for BufferSpec::FixedWidth.

get_byte_array_buffers 🔒

Returns the values and offsets [Buffer] for a ByteArray with offset type O

get_list_array_buffers 🔒

Similar logic as get_byte_array_buffers() but slices the child array instead of a values buffer.

get_list_view_array_buffers 🔒

Returns the offsets, sizes, and child data buffers for a ListView array.

get_or_truncate_buffer 🔒

Returns the sliced views [Buffer] for a BinaryView/Utf8View array.

has_validity_bitmap 🔒

In V4, null types have no validity bitmap In V5 and later, null and union types have no validity bitmap Run end encoded type has no validity bitmap.

into_zero_offset_run_array 🔒

pad_to_alignment 🔒

Calculate an alignment boundary and return the number of bytes needed to pad to the alignment boundary

reencode_offsets 🔒

Common functionality for re-encoding offsets. Returns the new offsets as well as original start offset and length for use in slicing child data.

unslice_run_array 🔒

write_array_data 🔒

Recursively encodes array_data into its IPC representation.

write_body_buffers 🔒

write_continuation 🔒

Write a record batch to the writer, writing the message size before the message if the record batch is being written to a stream

write_message

Write a message’s IPC data and buffers, returning metadata and buffer data lengths written