Struct IpcDataGenerator

pub struct IpcDataGenerator {}

Handles low level details of encoding [Array] and [Schema] into the Arrow IPC Format.

Example:

// Create a record batch
let batch = RecordBatch::try_from_iter(vec![
 ("col2", Arc::new(UInt64Array::from_iter([10, 23, 33])) as _)
]).unwrap();

// Error of dictionary ids are replaced.
let error_on_replacement = true;
let options = IpcWriteOptions::default();
let mut dictionary_tracker = DictionaryTracker::new(error_on_replacement);

// encode the batch into zero or more encoded dictionaries
// and the data for the actual array.
let data_gen = IpcDataGenerator::default();
let (encoded_dictionaries, encoded_message) = data_gen
  .encoded_batch(&batch, &mut dictionary_tracker, &options)
  .unwrap();

Implementations

impl IpcDataGenerator

pub fn schema_to_bytes_with_dictionary_tracker( &self, schema: &Schema, dictionary_tracker: &mut DictionaryTracker, write_options: &IpcWriteOptions, ) -> EncodedData

Converts a schema to an IPC message along with dictionary_tracker and returns it encoded inside EncodedData as a flatbuffer

Preferred method over IpcDataGenerator::schema_to_bytes since it’s deprecated since Arrow v54.0.0

pub fn schema_to_bytes( &self, schema: &Schema, write_options: &IpcWriteOptions, ) -> EncodedData

👎Deprecated since 54.0.0: Use schema_to_bytes_with_dictionary_tracker instead. This function signature of schema_to_bytes_with_dictionary_tracker in the next release.

Converts a schema to an IPC message and returns it encoded inside EncodedData as a flatbuffer

fn _encode_dictionaries<I: Iterator<Item = i64>>( &self, column: &ArrayRef, encoded_dictionaries: &mut Vec<EncodedData>, dictionary_tracker: &mut DictionaryTracker, write_options: &IpcWriteOptions, dict_id: &mut I, ) -> Result<(), ArrowError>

fn encode_dictionaries<I: Iterator<Item = i64>>( &self, field: &Field, column: &ArrayRef, encoded_dictionaries: &mut Vec<EncodedData>, dictionary_tracker: &mut DictionaryTracker, write_options: &IpcWriteOptions, dict_id_seq: &mut I, ) -> Result<(), ArrowError>

pub fn encoded_batch( &self, batch: &RecordBatch, dictionary_tracker: &mut DictionaryTracker, write_options: &IpcWriteOptions, ) -> Result<(Vec<EncodedData>, EncodedData), ArrowError>

Encodes a batch to a number of EncodedData items (dictionary batches + the record batch). The DictionaryTracker keeps track of dictionaries with new dict_ids (so they are only sent once) Make sure the DictionaryTracker is initialized at the start of the stream.

fn record_batch_to_bytes( &self, batch: &RecordBatch, write_options: &IpcWriteOptions, ) -> Result<EncodedData, ArrowError>

Write a RecordBatch into two sets of bytes, one for the header (crate::Message) and the other for the batch’s data

fn dictionary_batch_to_bytes( &self, dict_id: i64, array_data: &ArrayData, write_options: &IpcWriteOptions, ) -> Result<EncodedData, ArrowError>

Write dictionary values into two sets of bytes, one for the header (crate::Message) and the other for the data

Trait Implementations

impl Debug for IpcDataGenerator

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Default for IpcDataGenerator

fn default() -> IpcDataGenerator

Returns the “default value” for a type.