Struct IpcDataGenerator 

pub struct IpcDataGenerator {}

Handles low level details of encoding [Array] and [Schema] into the Arrow IPC Format.

Example

// Create a record batch
let batch = RecordBatch::try_from_iter(vec![
 ("col2", Arc::new(UInt64Array::from_iter([10, 23, 33])) as _)
]).unwrap();

// Error of dictionary ids are replaced.
let error_on_replacement = true;
let options = IpcWriteOptions::default();
let mut dictionary_tracker = DictionaryTracker::new(error_on_replacement);

let mut ipc_write_context = IpcWriteContext::default();

// encode the batch into zero or more encoded dictionaries
// and the data for the actual array.
let data_gen = IpcDataGenerator::default();
let (encoded_dictionaries, encoded_message) = data_gen
  .encode(&batch, &mut dictionary_tracker, &options, &mut ipc_write_context)
  .unwrap();

Implementations

impl IpcDataGenerator

pub fn schema_to_bytes_with_dictionary_tracker( &self, schema: &Schema, dictionary_tracker: &mut DictionaryTracker, write_options: &IpcWriteOptions, ) -> EncodedData

Converts a schema to an IPC message along with dictionary_tracker and returns it encoded inside EncodedData as a flatbuffer.

fn _collect_dict_updates<I: Iterator<Item = i64>>( &self, column: &ArrayRef, dictionaries_to_encode: &mut Vec<DictionaryToEncode>, dictionary_tracker: &mut DictionaryTracker, write_options: &IpcWriteOptions, dict_id: &mut I, ) -> Result<(), ArrowError>

fn collect_dict_updates<I: Iterator<Item = i64>>( &self, field: &Field, column: &ArrayRef, dictionaries_to_encode: &mut Vec<DictionaryToEncode>, dictionary_tracker: &mut DictionaryTracker, write_options: &IpcWriteOptions, dict_id_seq: &mut I, ) -> Result<(), ArrowError>

pub fn encode( &self, batch: &RecordBatch, dictionary_tracker: &mut DictionaryTracker, write_options: &IpcWriteOptions, ipc_write_context: &mut IpcWriteContext, ) -> Result<(Vec<EncodedData>, EncodedData), ArrowError>

Encodes a batch to a number of EncodedData items (dictionary batches + the record batch). The DictionaryTracker keeps track of dictionaries with new dict_ids (so they are only sent once) Make sure the DictionaryTracker is initialized at the start of the stream.

fn collect_all_dicts( &self, batch: &RecordBatch, dictionary_tracker: &mut DictionaryTracker, write_options: &IpcWriteOptions, ) -> Result<Vec<DictionaryToEncode>, ArrowError>

Walk the record batch and collect dictionaries for later encoding.

fn write<W: Write>( &self, batch: &RecordBatch, dictionary_tracker: &mut DictionaryTracker, write_options: &IpcWriteOptions, ipc_write_context: &mut IpcWriteContext, writer: &mut W, ) -> Result<IpcWriteMetadata, ArrowError>

Write dictionary batches and the record batch directly to writer, skipping the intermediate body Vec<u8> allocations Returns IpcWriteMetadata with the sizes needed to build footer blocks.

fn encode_to_buffers( &self, batch: &RecordBatch, dictionary_tracker: &mut DictionaryTracker, write_options: &IpcWriteOptions, ipc_write_context: &mut IpcWriteContext, out: &mut Vec<Buffer>, ) -> Result<IpcWriteMetadata, ArrowError>

Encode dictionary batches and the record batch to output buffers, skipping the intermediate body Vec<u8> allocations for uncompressed record batch buffers.

fn write_to_sink<S: IpcRecordBatchSink>( &self, batch: &RecordBatch, dictionary_tracker: &mut DictionaryTracker, write_options: &IpcWriteOptions, ipc_write_context: &mut IpcWriteContext, sink: &mut S, ) -> Result<IpcWriteMetadata, ArrowError>

pub fn encoded_batch( &self, batch: &RecordBatch, dictionary_tracker: &mut DictionaryTracker, write_options: &IpcWriteOptions, ) -> Result<(Vec<EncodedData>, EncodedData), ArrowError>

👎Deprecated since 57.0.0:

Use encode instead

Encodes a batch to a number of EncodedData items (dictionary batches + the record batch). The DictionaryTracker keeps track of dictionaries with new dict_ids (so they are only sent once) Make sure the DictionaryTracker is initialized at the start of the stream.

fn record_batch_to_bytes( &self, batch: &RecordBatch, write_options: &IpcWriteOptions, ipc_write_context: &mut IpcWriteContext, sink: &mut IpcBodySink<'_>, ) -> Result<usize, ArrowError>

Encodes a RecordBatch into a flatbuffer IPC message and fills sink with the serialised buffer data.

Returns the total body length written to sink (including per-buffer alignment padding).

The FlatBuffer crate::Message is located in ipc_write_context’s [FlatBufferBuilder] finished bytes. A successful Result from this function guarantees the builder is in a finished state to call [FlatBufferBuilder::finished_data].

fn dictionary_batch_to_bytes( &self, dict: DictionaryToEncode, write_options: &IpcWriteOptions, ipc_write_context: &mut IpcWriteContext, ) -> Result<EncodedData, ArrowError>

Write dictionary values into two sets of bytes, one for the header (crate::Message) and the other for the data

fn dictionary_batch_to_sink( &self, dict: &DictionaryToEncode, write_options: &IpcWriteOptions, ipc_write_context: &mut IpcWriteContext, sink: &mut IpcBodySink<'_>, ) -> Result<usize, ArrowError>

Encodes a dictionary batch’s flatbuffer metadata into ipc_write_context’s builder (read back via ipc_write_context.mut_fbb().finished_data()) and fills sink with its body buffers.

Returns the total body length written to sink (including per-buffer alignment padding).

fn encode_record_batch_data( &self, columns: impl IntoIterator<Item = ArrayData>, row_count: i64, write_options: &IpcWriteOptions, ipc_write_context: &mut IpcWriteContext, sink: &mut IpcBodySink<'_>, ) -> Result<EncodedRecordBatchMeta, ArrowError>

Encode the body buffers of a single record batch into sink and build the flatbuffer RecordBatch table into ipc_write_context’s builder.

This is the shared core of Self::record_batch_to_bytes and Self::dictionary_batch_to_sink: a dictionary message embeds a record batch (its single column of values), so both encode the same RecordBatch table and only differ in how they wrap the returned offset in a message.

columns yields the column data in IPC buffer order and row_count is the logical length recorded in the RecordBatch header. The caller is responsible for wrapping the returned EncodedRecordBatchMeta::fb_offset in a Message and calling [FlatBufferBuilder::finish].

Trait Implementations

impl Debug for IpcDataGenerator

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Default for IpcDataGenerator

fn default() -> Self

Returns the “default value” for a type.