Struct ParquetMetaDataPushDecoder 

pub struct ParquetMetaDataPushDecoder {
    state: DecodeState,
    column_index_policy: PageIndexPolicy,
    offset_index_policy: PageIndexPolicy,
    column_index_mask: ColumnChunkMask,
    offset_index_mask: ColumnChunkMask,
    buffers: PushBuffers,
    metadata_parser: MetadataParser,
}

A push decoder for ParquetMetaData.

This structure implements a push API for decoding Parquet metadata, which decouples IO from the metadata decoding logic (sometimes referred to as Sans-IO).

See ParquetMetaDataReader for a pull-based API that incorporates IO and is simpler to use for basic use cases. This decoder is best for customizing your IO operations to minimize bytes read, prefetch data, or use async IO.

Example

The most basic usage is to feed the decoder with the necessary byte ranges as requested as shown below. This minimizes the number of bytes read, but requires the most IO operations - one to read the footer and then one to read the metadata, and possibly more if page indexes are requested.

// The `ParquetMetaDataPushDecoder` needs to know the file length.
let mut decoder = ParquetMetaDataPushDecoder::try_new(file_len).unwrap();
// try to decode the metadata. If more data is needed, the decoder will tell you what ranges
loop {
    match decoder.try_decode() {
       Ok(DecodeResult::Data(metadata)) => { return Ok(metadata); } // decode successful
       Ok(DecodeResult::NeedsData(ranges)) => {
          // The decoder needs more data
          //
          // In this example, we call a function that returns the bytes for each given range.
          // In a real application, you would likely read the data from a file or network.
          let data = ranges.iter().map(|range| get_range(range)).collect();
          // Push the data into the decoder and try to decode again on the next iteration.
          decoder.push_ranges(ranges, data).unwrap();
       }
       Ok(DecodeResult::Finished) => { unreachable!("returned metadata in previous match arm") }
       Err(e) => return Err(e),
    }
}

Example with “prefetching”

By default, the ParquetMetaDataPushDecoder will request only the exact byte ranges it needs. This minimizes the number of bytes read, however it requires at least two IO operations to read the metadata - one to read the footer and then one to read the metadata.

If the file has a “Page Index” (see Self::with_page_index_policy), three IO operations are required to read the metadata, as the page index is not part of the normal metadata footer.

To reduce the number of IO operations in systems with high per operation overhead (e.g. cloud storage), you can “prefetch” the data and then push the data into the decoder before calling Self::try_decode. If you do not push enough bytes, the decoder will return the ranges that are still needed.

This approach can also be used when you have the entire file already in memory for other reasons.

let file_len = file_bytes.len() as u64;
// For this example, we "prefetch" all the bytes which we have in memory,
// but in a real application, you would likely read a chunk from the end
// for example 1MB.
let prefetched_bytes = file_bytes.clone();
let mut decoder = ParquetMetaDataPushDecoder::try_new(file_len).unwrap();
// push the prefetched bytes into the decoder
decoder.push_ranges(vec![0..file_len], vec![prefetched_bytes]).unwrap();
// The decoder will now be able to decode the metadata. Note in a real application,
// unless you can guarantee that the pushed data is enough to decode the metadata,
// you still need to call `try_decode` in a loop until it returns `DecodeResult::Data`
// as shown in  the previous example
    match decoder.try_decode() {
        Ok(DecodeResult::Data(metadata)) => { return Ok(metadata); } // decode successful
        other => { panic!("expected DecodeResult::Data, got: {other:?}") }
    }

Example using AsyncRead

ParquetMetaDataPushDecoder is designed to work with any data source that can provide byte ranges, including async IO sources. However, it does not implement async IO itself. To use async IO, you simply write an async wrapper around it that reads the required byte ranges and pushes them into the decoder.

use tokio::io::{AsyncRead, AsyncReadExt, AsyncSeek, AsyncSeekExt};
// This function decodes Parquet Metadata from anything that implements
// [`AsyncRead`] and [`AsyncSeek`] such as a tokio::fs::File
async fn decode_metadata(
  file_len: u64,
  mut async_source: impl AsyncRead + AsyncSeek + Unpin
) -> Result<ParquetMetaData, ParquetError> {
  // We need a ParquetMetaDataPushDecoder to decode the metadata.
  let mut decoder = ParquetMetaDataPushDecoder::try_new(file_len).unwrap();
  loop {
    match decoder.try_decode() {
       Ok(DecodeResult::Data(metadata)) => { return Ok(metadata); } // decode successful
       Ok(DecodeResult::NeedsData(ranges)) => {
          // The decoder needs more data
          //
          // In this example we use the AsyncRead and AsyncSeek traits to read the
          // required ranges from the async source.
          let mut data = Vec::with_capacity(ranges.len());
          for range in &ranges {
            let mut buffer = vec![0; (range.end - range.start) as usize];
            async_source.seek(std::io::SeekFrom::Start(range.start)).await?;
            async_source.read_exact(&mut buffer).await?;
            data.push(Bytes::from(buffer));
          }
          // Push the data into the decoder and try to decode again on the next iteration.
          decoder.push_ranges(ranges, data).unwrap();
       }
       Ok(DecodeResult::Finished) => { unreachable!("returned metadata in previous match arm") }
       Err(e) => return Err(e),
    }
  }
}

Fields

state: DecodeState

Decoding state

column_index_policy: PageIndexPolicy

policy for loading ColumnIndex (part of the PageIndex)

offset_index_policy: PageIndexPolicy

policy for loading OffsetIndex (part of the PageIndex)

column_index_mask: ColumnChunkMask

which rows and columns of the column index should be read

offset_index_mask: ColumnChunkMask

which rows and columns of the offset index should be read

buffers: PushBuffers

Underlying buffers

metadata_parser: MetadataParser

Encryption API

Implementations

impl ParquetMetaDataPushDecoder

pub fn try_new(file_len: u64) -> Result<Self>

Create a new ParquetMetaDataPushDecoder with the given file length.

By default, this will read page indexes and column indexes. See ParquetMetaDataPushDecoder::with_page_index_policy for more detail.

See examples on ParquetMetaDataPushDecoder.

pub(crate) fn try_new_with_footer_tail( file_len: u64, footer_tail: FooterTail, ) -> Result<Self>

Begin decoding from the given footer tail.

pub fn try_new_with_metadata( file_len: u64, metadata: ParquetMetaData, ) -> Result<Self>

Create a decoder with the given ParquetMetaData already known.

This can be used to parse and populate the page index structures after the metadata has already been decoded.

The page indexes can then be decoded in one of two ways:

• Self::try_decode attaches the decoded page indexes to the returned metadata, replacing (or clearing) any existing page indexes See ParquetMetaDataReader::read_page_indexes for details.
• Self::try_decode_page_index returns the decoded page indexes separately and leaves any page indexes already attached to metadata unchanged.

pub fn with_page_index_policy(self, page_index_policy: PageIndexPolicy) -> Self

Enable or disable reading the page index structures described in “Parquet page index Layout to Support Page Skipping”.

Defaults to PageIndexPolicy::Optional

This requires

1. The Parquet file to have been written with page indexes
2. Additional data to be pushed into the decoder (as the page indexes are not part of the thrift footer)

pub fn with_column_index_policy( self, column_index_policy: PageIndexPolicy, ) -> Self

Set the policy for reading the ColumnIndex (part of the PageIndex)

pub fn with_offset_index_policy( self, offset_index_policy: PageIndexPolicy, ) -> Self

Set the policy for reading the OffsetIndex (part of the PageIndex)

pub fn with_page_index_mask(self, mask: ColumnChunkMask) -> Self

Select the row groups and columns for which both page index structures are read.

pub fn with_column_index_mask(self, mask: ColumnChunkMask) -> Self

Select the row groups and columns for which column indexes are read.

pub fn with_offset_index_mask(self, mask: ColumnChunkMask) -> Self

Select the row groups and columns for which offset indexes are read.

pub fn with_metadata_options( self, options: Option<Arc<ParquetMetaDataOptions>>, ) -> Self

Set the options to use when decoding the Parquet metadata.

pub fn with_file_decryption_properties( self, file_decryption_properties: Option<Arc<FileDecryptionProperties>>, ) -> Self

Provide decryption properties for decoding encrypted Parquet files

pub fn push_ranges( &mut self, ranges: Vec<Range<u64>>, buffers: Vec<Bytes>, ) -> Result<()>

Push the data into the decoder’s buffer.

The decoder does not immediately attempt to decode the metadata after pushing data. Instead, it accumulates the pushed data until you call Self::try_decode.

Determining required data:

To determine what ranges are required to decode the metadata, you can either:

1. Call Self::try_decode first to get the exact ranges required (see example on Self)

2. Speculatively push any data that you have available, which may include more than the footer data or requested bytes.

Speculatively pushing data can be used when “prefetching” data. See example on Self

pub fn push_range(&mut self, range: Range<u64>, buffer: Bytes) -> Result<()>

Pushes a single range of data into the decoder’s buffer.

pub fn clear_all_ranges(&mut self)

Clear any staged byte ranges currently buffered for future decode work.

pub fn try_decode(&mut self) -> Result<DecodeResult<ParquetMetaData>>

Try to decode the metadata from the pushed data, returning the decoded metadata or an error if not enough data is available.

pub fn try_decode_page_index( &mut self, ) -> Result<DecodeResult<(ParquetMetaData, Option<PageIndex>)>>

Try to decode the page indexes from the pushed data, returning the stored ParquetMetaData together with the decoded PageIndex.

This method can only be called once the footer metadata is known: either after Self::try_new_with_metadata, or after Self::try_decode has decoded the footer metadata and requested page index data. Calling it earlier returns an error and leaves the decoder unchanged.

This method is useful when you want to:

1. First decode the footer metadata without reading the page indexes (using PageIndexPolicy::Skip)
2. Later selectively decode a subset of the page indexes using Self::with_column_index_mask or Self::with_offset_index_mask

When the data can be accessed through a ChunkReader, prefer ParquetMetaDataReader::read_page_index_and_finish (or ParquetMetaDataReader::load_page_index_and_finish for asynchronous I/O), which handles the decoder state machine and I/O requests. See the read_page_index_and_finish example for the complete two-stage workflow.

Returns

• Ok(DecodeResult::Data((metadata, Some(page_index)))) - Successfully decoded the requested page indexes based on the configured policies and masks. The newly decoded page index is returned separately; any page index already attached to metadata is unchanged.
• Ok(DecodeResult::Data((metadata, None))) - No page indexes were requested or found according to the configured policies and masks. The stored metadata is still returned.
• Ok(DecodeResult::NeedsData(ranges)) - More data is needed to decode the page indexes. Push the requested ranges and call this method again.
• Ok(DecodeResult::Finished) - The decoder has finished and no more data can be decoded
• Err(_) - An error occurred during decoding

fn try_decode_page_index_for_metadata( &mut self, metadata: Box<ParquetMetaData>, ) -> Result<DecodeResult<(Box<ParquetMetaData>, Option<PageIndex>)>>

fn get_bytes(&self, range: &Range<u64>) -> Result<Bytes>

Returns the bytes for the given range from the internal buffer

Trait Implementations

impl Debug for ParquetMetaDataPushDecoder

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.