Struct StreamEncoder 

pub struct StreamEncoder {
    schema: Schema,
    write_options: IpcWriteOptions,
    schema_encoded: bool,
    dictionary_tracker: DictionaryTracker,
    data_gen: IpcDataGenerator,
    ipc_write_context: IpcWriteContext,
}

Arrow IPC stream encoder.

Encodes Arrow [RecordBatch]es to byte buffers using the [IPC Streaming Format], without performing any IO.

The returned [Buffer]s are ordered and should be written to the destination stream in order. Uncompressed record batch body buffers can share the original Arrow buffers instead of being copied into an intermediate contiguous buffer.

Example

let batch = record_batch!(("a", Int32, [1, 2, 3]))?;

let mut encoder = StreamEncoder::try_new(&batch.schema())?;
let mut stream = vec![];
for buffer in encoder.encode(&batch)? {
    stream.extend_from_slice(buffer.as_slice());
}
for buffer in encoder.finish()? {
    stream.extend_from_slice(buffer.as_slice());
}

Fields

schema: Schema

write_options: IpcWriteOptions

IPC write options

schema_encoded: bool

Whether the stream schema has been encoded

dictionary_tracker: DictionaryTracker

Keeps track of dictionaries that have been encoded

data_gen: IpcDataGenerator

ipc_write_context: IpcWriteContext

Implementations

impl StreamEncoder

pub fn try_new(schema: &Schema) -> Result<Self, ArrowError>

Try to create a new stream encoder.

pub fn try_new_with_options( schema: &Schema, write_options: IpcWriteOptions, ) -> Result<Self, ArrowError>

Try to create a new stream encoder with IpcWriteOptions.

pub fn encode(&mut self, batch: &RecordBatch) -> Result<Vec<Buffer>, ArrowError>

Encode a [RecordBatch] into buffers.

The first call also includes the IPC stream schema message before the record batch message. Later calls only include dictionary and record batch messages.

Errors

Returns an error if encoding fails.

pub fn finish(self) -> Result<Vec<Buffer>, ArrowError>

Encode the end-of-stream marker.

If no batches have been encoded, this also emits the IPC stream schema message so the returned buffers form a valid empty IPC stream.

Errors

Returns an error if encoding the schema or end-of-stream marker fails.

fn encode_schema(&mut self, out: &mut Vec<Buffer>) -> Result<(), ArrowError>