Struct ParquetRecordBatchStream

pub struct ParquetRecordBatchStream<T> {
    metadata: Arc<ParquetMetaData>,
    schema: SchemaRef,
    row_groups: VecDeque<usize>,
    projection: ProjectionMask,
    batch_size: usize,
    selection: Option<RowSelection>,
    reader: Option<ReaderFactory<T>>,
    state: StreamState<T>,
}

An asynchronous Streamof [RecordBatch] constructed using ParquetRecordBatchStreamBuilder to read parquet files.

ParquetRecordBatchStream also provides ParquetRecordBatchStream::next_row_group for fetching row groups, allowing users to decode record batches separately from I/O.

I/O Buffering

ParquetRecordBatchStream buffers all data pages selected after predicates (projection + filtering, etc) and decodes the rows from those buffered pages.

For example, if all rows and columns are selected, the entire row group is buffered in memory during decode. This minimizes the number of IO operations required, which is especially important for object stores, where IO operations have latencies in the hundreds of milliseconds

Fields

metadata: Arc<ParquetMetaData>

schema: SchemaRef

row_groups: VecDeque<usize>

projection: ProjectionMask

batch_size: usize

selection: Option<RowSelection>

reader: Option<ReaderFactory<T>>

This is an option so it can be moved into a future

state: StreamState<T>

Implementations

impl<T> ParquetRecordBatchStream<T>

pub fn schema(&self) -> &SchemaRef

Returns the projected [SchemaRef] for reading the parquet file.

Note that the schema metadata will be stripped here. See ParquetRecordBatchStreamBuilder::schema if the metadata is desired.

impl<T> ParquetRecordBatchStream<T>

where T: AsyncFileReader + Unpin + Send + 'static,

pub async fn next_row_group( &mut self, ) -> Result<Option<ParquetRecordBatchReader>>

Fetches the next row group from the stream.

Users can continue to call this function to get row groups and decode them concurrently.

Notes

ParquetRecordBatchStream should be used either as a Stream or with next_row_group; they should not be used simultaneously.

Returns

• Ok(None) if the stream has ended.
• Err(error) if the stream has errored. All subsequent calls will return Ok(None).
• Ok(Some(reader)) which holds all the data for the row group.

Trait Implementations

impl<T> Debug for ParquetRecordBatchStream<T>

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl<T> Stream for ParquetRecordBatchStream<T>

where T: AsyncFileReader + Unpin + Send + 'static,

type Item = Result<RecordBatch, ParquetError>

Values yielded by the stream.

fn poll_next( self: Pin<&mut Self>, cx: &mut Context<'_>, ) -> Poll<Option<Self::Item>>

Attempt to pull out the next value of this stream, registering the current task for wakeup if the value is not yet available, and returning None if the stream is exhausted.

fn size_hint(&self) -> (usize, Option<usize>)

Returns the bounds on the remaining length of the stream.