Module writer

Arrow IPC File and Stream Writers

Notes

FileWriter and StreamWriter have similar interfaces, however the FileWriter expects a reader that supports Seeking

Structs

DictionaryTracker

Keeps track of dictionaries that have been written, to avoid emitting the same dictionary multiple times.

EncodedData

Stores the encoded data, which is an crate::Message, and optional Arrow data

FileWriter

Arrow File Writer

IpcDataGenerator

Handles low level details of encoding [Array] and [Schema] into the Arrow IPC Format.

IpcWriteOptions

IPC write options used to control the behaviour of the IpcDataGenerator

StreamWriter

Arrow Stream Writer

Constants

PADDING 🔒

Functions

append_variadic_buffer_counts 🔒

buffer_need_truncate 🔒

Whether to truncate the buffer

get_buffer_element_width 🔒

Returns byte width for a buffer spec. Only for BufferSpec::FixedWidth.

get_byte_array_buffers 🔒

Returns the values and offsets [Buffer] for a ByteArray with offset type O

get_list_array_buffers 🔒

Similar logic as get_byte_array_buffers() but slices the child array instead of a values buffer.

has_validity_bitmap 🔒

In V4, null types have no validity bitmap In V5 and later, null and union types have no validity bitmap Run end encoded type has no validity bitmap.

into_zero_offset_run_array 🔒

pad_to_alignment 🔒

Calculate an alignment boundary and return the number of bytes needed to pad to the alignment boundary

reencode_offsets 🔒

Common functionality for re-encoding offsets. Returns the new offsets as well as original start offset and length for use in slicing child data.

unslice_run_array 🔒

write_array_data 🔒

Write array data to a vector of bytes

write_body_buffers 🔒

write_buffer 🔒

Write a buffer into arrow_data, a vector of bytes, and adds its crate::Buffer to buffers. Returns the new offset in arrow_data

write_continuation 🔒

Write a record batch to the writer, writing the message size before the message if the record batch is being written to a stream

write_message

Write a message’s IPC data and buffers, returning metadata and buffer data lengths written