Struct ParquetPushDecoder 

pub struct ParquetPushDecoder {
    state: ParquetDecoderState,
}

A push based Parquet Decoder

See ParquetPushDecoderBuilder for an example of how to build and use the decoder.

ParquetPushDecoder is a low level API for decoding Parquet data without an underlying reader for performing IO, and thus offers fine grained control over how data is fetched and decoded.

When more data is needed to make progress, instead of reading data directly from a reader, the decoder returns DecodeResult indicating what ranges are needed. Once the caller provides the requested ranges via Self::push_ranges, they try to decode again by calling Self::try_decode.

The decoder’s internal state tracks what has been already decoded and what is needed next.

Fields

state: ParquetDecoderState

The inner state.

This state is consumed on every transition and a new state is produced so the Rust compiler can ensure that the state is always valid and transitions are not missed.

Implementations

impl ParquetPushDecoder

pub fn try_decode(&mut self) -> Result<DecodeResult<RecordBatch>, ParquetError>

Attempt to decode the next batch of data, or return what data is needed

The the decoder communicates the next state with a DecodeResult

See full example in ParquetPushDecoderBuilder

use parquet::DecodeResult;
let mut decoder = get_decoder();
loop {
   match decoder.try_decode().unwrap() {
      DecodeResult::NeedsData(ranges) => {
        // The decoder needs more data. Fetch the data for the given ranges
        // call decoder.push_ranges(ranges, data) and call again
        push_data(&mut decoder, ranges);
      }
      DecodeResult::Data(batch) => {
        // Successfully decoded the next batch of data
        println!("Got batch with {} rows", batch.num_rows());
      }
      DecodeResult::Finished => {
        // The decoder has finished decoding all data
        break;
      }
   }
}

pub fn try_next_reader( &mut self, ) -> Result<DecodeResult<ParquetRecordBatchReader>, ParquetError>

Return a ParquetRecordBatchReader that reads the next set of rows, or return what data is needed to produce it.

This API can be used to get a reader for decoding the next set of RecordBatches while proceeding to begin fetching data for the set (e.g row group)

Example

use parquet::DecodeResult;
let mut decoder = get_decoder();
loop {
   match decoder.try_next_reader().unwrap() {
      DecodeResult::NeedsData(ranges) => {
        // The decoder needs more data. Fetch the data for the given ranges
        // call decoder.push_ranges(ranges, data) and call again
        push_data(&mut decoder, ranges);
      }
      DecodeResult::Data(reader) => {
         // spawn a thread to read the batches in parallel
         // with fetching the next row group / data
         std::thread::spawn(move || {
           for batch in reader {
             let batch = batch.unwrap();
             println!("Got batch with {} rows", batch.num_rows());
           }
        });
      }
      DecodeResult::Finished => {
        // The decoder has finished decoding all data
        break;
      }
   }
}

pub fn push_range( &mut self, range: Range<u64>, data: Bytes, ) -> Result<(), ParquetError>

Push data into the decoder for processing

This is a convenience wrapper around Self::push_ranges for pushing a single range of data.

Note this can be the entire file or just a part of it. If it is part of the file, the ranges should correspond to the data ranges requested by the decoder.

Memory Usage

The decoder releases the pushed bytes that it will not read again (for example, the bytes of a row group when it is done with that row group). However, it does not release all pushed bytes: it keeps the bytes of row groups that it does not read (for example, row groups without selected rows, or after the limit). Call Self::clear_all_ranges to release them.

See example in ParquetPushDecoderBuilder

pub fn push_ranges( &mut self, ranges: Vec<Range<u64>>, data: Vec<Bytes>, ) -> Result<(), ParquetError>

Push data into the decoder for processing

This should correspond to the data ranges requested by the decoder.

If you fetch data before the decoder requests it, for example with Self::scan_plan, keep it in your own cache and push exactly the ranges in each DecodeResult::NeedsData:

• The decoder does not use a requested range that is split over two pushed buffers.
• The decoder releases a pushed buffer only if its range is equal to a requested range. Other pushed data stays buffered.

pub fn buffered_bytes(&self) -> u64

Returns the total number of buffered bytes in the decoder

This is the sum of the size of all [Bytes] that has been pushed to the decoder but not yet consumed.

Note that this does not include any overhead of the internal data structures and that since [Bytes] are ref counted memory, this may not reflect additional memory usage.

This can be used to monitor memory usage of the decoder.

pub fn clear_all_ranges(&mut self)

Clear any staged byte ranges currently buffered for future decode work.

This clears byte ranges still owned by the decoder’s internal PushBuffers. It does not affect any data that has already been handed off to an active ParquetRecordBatchReader.

pub fn is_at_row_group_boundary(&self) -> bool

True iff the decoder is at a row-group boundary, where Self::into_builder can reconfigure the scan.

A boundary is “between row groups”: the previous row group’s ParquetRecordBatchReader has been fully extracted (via Self::try_next_reader) or fully drained (via Self::try_decode), and the next row group has not yet been planned. While Self::try_decode is iterating an active row group’s reader this returns false; with Self::try_next_reader there is a clean window between two consecutive returns where this is true.

pub fn row_groups_remaining(&self) -> usize

Number of row groups left to decode after the one currently in flight. Useful as a “should I bother reconfiguring the scan?” signal.

pub fn peek_next_row_group(&self) -> Result<Option<usize>, ParquetError>

Returns the row-group index that the next call to Self::try_next_reader will yield a reader for, after applying any internal skipping (row selection emptiness, exhausted budget, finished state).

Safe to call at any time. When called mid-row-group (i.e. while a previously-emitted ParquetRecordBatchReader is still being drained), the returned index refers to the row group that Self::try_next_reader will produce after the current one.

Returns Ok(None) when:

• the decoder has no more row groups to read, or
• every remaining row group would be skipped.

This method does not mutate decoder state. It is useful for callers that maintain per-row-group state in lock-step with the decoder (e.g. dynamic row-group pruners) to determine which row group the next reader corresponds to, since Self::try_next_reader may silently advance past row groups based on filtering and other criteria.

pub fn scan_plan(&self) -> ScanPlan

Returns the byte ranges that this decoder may still read, in the order that decoding needs them. See ScanPlan.

pub fn into_builder(self) -> Result<ParquetPushDecoderBuilder, ParquetError>

Decompose this decoder back into a ParquetPushDecoderBuilder for the row groups that have not yet been decoded.

This is the API for adaptive scans. Drive the decoder with Self::try_next_reader; at any row-group boundary, call into_builder to recover a builder, adjust it with the usual ParquetPushDecoderBuilder setters, and build a fresh decoder that resumes from the next row group:

let mut decoder = get_decoder();
// ... drive `decoder.try_next_reader()` for a few row groups ...
if decoder.is_at_row_group_boundary() && decoder.row_groups_remaining() > 0 {
    decoder = decoder
        .into_builder()
        .unwrap()
        // any builder option can be changed here, e.g. promote a
        // filter into a row filter based on observed selectivity
        .with_row_filter(new_filter())
        .build()
        .unwrap();
}

The returned builder preserves the not-yet-decoded row-group plan, including any row-group-local selections, together with the remaining offset/limit budget. Rows from already-decoded row groups are not produced again. Every other option — projection, row filter, row selection policy, batch size, metrics, predicate-cache size — is left exactly as the decoder had it and can be overridden before build.

Because the preserved plan pins the remaining row groups, the mutual exclusion documented on with_row_group_selections applies to the rebuilt builder as well: calling it on a builder rebuilt from a decoder configured with with_row_groups / with_row_selection (or vice versa) returns an error from build.

Errors

Returns Err(ParquetError::General) when the decoder is not at a row-group boundary (check Self::is_at_row_group_boundary first) or has already finished. The decoder is consumed either way.

Buffered bytes

The decoder’s buffered bytes are carried across the rebuild: bytes already fetched for row groups the new configuration still reads are not re-requested. build releases the bytes that the new configuration does not read. See push_range for more details.

Trait Implementations

impl Debug for ParquetPushDecoder

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.