Struct StreamDecoder 

pub struct StreamDecoder {
    schema: Option<SchemaRef>,
    dictionaries: HashMap<i64, ArrayRef>,
    state: DecoderState,
    buf: MutableBuffer,
    require_alignment: bool,
    skip_validation: UnsafeFlag,
}

A low-level interface for reading [RecordBatch] data from a stream of bytes

See StreamReader for a higher-level interface

Fields

schema: Option<SchemaRef>

The schema of this decoder, if read

dictionaries: HashMap<i64, ArrayRef>

Lookup table for dictionaries by ID

state: DecoderState

The decoder state

buf: MutableBuffer

A scratch buffer when a read is split across multiple Buffer

require_alignment: bool

Whether or not array data in input buffers are required to be aligned

skip_validation: UnsafeFlag

Should validation be skipped when reading data? Defaults to false.

See FileDecoder::with_skip_validation for details.

Implementations

impl StreamDecoder

pub fn new() -> Self

Create a new StreamDecoder

pub fn with_require_alignment(self, require_alignment: bool) -> Self

Specifies whether or not array data in input buffers is required to be properly aligned.

If require_alignment is true, this decoder will return an error if any array data in the input buf is not properly aligned. Under the hood it will use [arrow_data::ArrayDataBuilder::build] to construct [arrow_data::ArrayData].

If require_alignment is false (the default), this decoder will automatically allocate a new aligned buffer and copy over the data if any array data in the input buf is not properly aligned. (Properly aligned array data will remain zero-copy.) Under the hood it will use [arrow_data::ArrayDataBuilder::build_aligned] to construct [arrow_data::ArrayData].

pub fn schema(&self) -> Option<SchemaRef>

Return the schema if decoded, else None.

pub fn decode( &mut self, buffer: &mut Buffer, ) -> Result<Option<RecordBatch>, ArrowError>

Try to read the next [RecordBatch] from the provided [Buffer]

[Buffer::advance] will be called on buffer for any consumed bytes.

The push-based interface facilitates integration with sources that yield arbitrarily delimited bytes ranges, such as a chunked byte stream received from object storage

fn print_stream<I>(src: impl Iterator<Item = Buffer>) -> Result<(), ArrowError> {
    let mut decoder = StreamDecoder::new();
    for mut x in src {
        while !x.is_empty() {
            if let Some(x) = decoder.decode(&mut x)? {
                println!("{x:?}");
            }
            if let Some(schema) = decoder.schema() {
                println!("Schema: {schema:?}");
            }
        }
    }
    decoder.finish().unwrap();
    Ok(())
}

pub fn finish(&mut self) -> Result<(), ArrowError>

Signal the end of stream

Returns an error if any partial data remains in the stream

Trait Implementations

impl Debug for StreamDecoder

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Default for StreamDecoder

fn default() -> StreamDecoder

Returns the “default value” for a type.