Struct PushBuffers 

pub struct PushBuffers {
    offset: u64,
    file_len: u64,
    ranges: Vec<Range<u64>>,
    buffers: Vec<Bytes>,
}

Holds multiple non-contiguous, caller-provided buffers of file data.

This is the in-memory buffer used by the push-based Parquet decoders (ParquetPushDecoder and ParquetMetaDataPushDecoder). It can be constructed up front and handed to a builder so the decoder reuses bytes that have already been fetched.

Features:

1. Zero copy
2. non contiguous ranges of bytes

Non Coalescing

This buffer does not coalesce (merging adjacent ranges of bytes into a single range). Coalescing at this level would require copying the data but the caller may already have the needed data in a single buffer which would require no copying.

Thus, the implementation defers to the caller to coalesce subsequent requests if desired.

Fields

offset: u64

the virtual “offset” of this buffers (added to any request)

file_len: u64

The total length of the file being decoded

ranges: Vec<Range<u64>>

The ranges of data that are available for decoding (not adjusted for offset)

buffers: Vec<Bytes>

The buffers of data that can be used to decode the Parquet file

Implementations

impl PushBuffers

pub fn new(file_len: u64) -> Self

Create a new, empty PushBuffers for a file of the given length.

Use PushBuffers::default when the file length is unknown or irrelevant (e.g. the push decoder, which tracks ranges by absolute offset and never consults file_len).

pub fn push_ranges( &mut self, ranges: Vec<Range<u64>>, buffers: Vec<Bytes>, ) -> Result<(), ParquetError>

Push all the ranges and buffers

Errors

Returns an error if the number of ranges does not match the number of buffers, or if any buffer’s length does not match its range (see Self::push_range).

pub fn push_range( &mut self, range: Range<u64>, buffer: Bytes, ) -> Result<(), ParquetError>

Push a new range and its associated buffer

Errors

Returns an error if the buffer’s length does not match the range’s length, e.g. when a truncated (short) read is pushed.

pub(crate) fn has_range(&self, range: &Range<u64>) -> bool

Returns true if the Buffers contains data for the given range

fn iter(&self) -> impl Iterator<Item = (&Range<u64>, &Bytes)>

pub(crate) fn file_len(&self) -> u64

return the file length of the Parquet file being read

fn with_offset(self, offset: u64) -> Self

Specify a new offset

pub(crate) fn buffered_bytes(&self) -> u64

Return the total of all buffered ranges

pub(crate) fn clear_ranges(&mut self, ranges_to_clear: &[Range<u64>])

Clear any range and corresponding buffer that is exactly in the ranges_to_clear

pub(crate) fn clear_all_ranges(&mut self)

Clear all buffered ranges and their corresponding data

Trait Implementations

impl ChunkReader for PushBuffers

type T = PushBuffers

The concrete type of reader returned by this trait

fn get_read(&self, start: u64) -> Result<Self::T, ParquetError>

Get a Read instance starting at the provided file offset

fn get_bytes(&self, start: u64, length: usize) -> Result<Bytes, ParquetError>

Get a range of data in memory as [Bytes]

impl Clone for PushBuffers

fn clone(&self) -> PushBuffers

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for PushBuffers

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Default for PushBuffers

fn default() -> PushBuffers

Returns the “default value” for a type.

impl Display for PushBuffers

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Length for PushBuffers

fn len(&self) -> u64

Returns the amount of bytes of the inner source.

impl Read for PushBuffers

less efficient implementation of Read for Buffers

fn read(&mut self, buf: &mut [u8]) -> Result<usize>

Pull some bytes from this source into the specified buffer, returning how many bytes were read.

fn read_vectored(&mut self, bufs: &mut [IoSliceMut<'_>]) -> Result<usize, Error>

Like read, except that it reads into a slice of buffers.

fn is_read_vectored(&self) -> bool

🔬This is a nightly-only experimental API. (can_vector)

Determines if this Reader has an efficient read_vectored implementation.

fn read_to_end(&mut self, buf: &mut Vec<u8>) -> Result<usize, Error>

Reads all bytes until EOF in this source, placing them into buf.

fn read_to_string(&mut self, buf: &mut String) -> Result<usize, Error>

Reads all bytes until EOF in this source, appending them to buf.

fn read_exact(&mut self, buf: &mut [u8]) -> Result<(), Error>

Reads the exact number of bytes required to fill buf.

fn read_buf(&mut self, buf: BorrowedCursor<'_, u8>) -> Result<(), Error>

🔬This is a nightly-only experimental API. (read_buf)

Pull some bytes from this source into the specified buffer.

fn read_buf_exact( &mut self, cursor: BorrowedCursor<'_, u8>, ) -> Result<(), Error>

🔬This is a nightly-only experimental API. (read_buf)

Reads the exact number of bytes required to fill cursor.

fn by_ref(&mut self) -> &mut Self

where Self: Sized,

Creates a “by reference” adapter for this instance of Read.

fn bytes(self) -> Bytes<Self>

where Self: Sized,

Transforms this Read instance to an Iterator over its bytes.

fn chain<R>(self, next: R) -> Chain<Self, R>

where R: Read, Self: Sized,

Creates an adapter which will chain this stream with another.

fn take(self, limit: u64) -> Take<Self>

where Self: Sized,

Creates an adapter which will read at most limit bytes from it.

fn read_array<const N: usize>(&mut self) -> Result<[u8; N], Error>

where Self: Sized,

🔬This is a nightly-only experimental API. (read_array)

Read and return a fixed array of bytes from this source.

fn read_le<T>(&mut self) -> Result<T, Error>

where T: FromEndianBytes, Self: Sized,

🔬This is a nightly-only experimental API. (read_le)

Read and return a type (e.g. an integer) in little-endian order.

fn read_be<T>(&mut self) -> Result<T, Error>

where T: FromEndianBytes, Self: Sized,

🔬This is a nightly-only experimental API. (read_le)

Read and return a type (e.g. an integer) in big-endian order.