Struct PushBuffers 

pub struct PushBuffers {
    offset: u64,
    file_len: u64,
    ranges: Vec<Range<u64>>,
    buffers: Vec<Bytes>,
}

Holds multiple non-contiguous, caller-provided buffers of file data.

This is the in-memory buffer used by the push-based Parquet decoders (ParquetPushDecoder and ParquetMetaDataPushDecoder). It can be constructed up front and handed to a builder so the decoder reuses bytes that have already been fetched.

Features:

1. Zero copy
2. non contiguous ranges of bytes

Non Coalescing

This buffer does not coalesce (merging adjacent ranges of bytes into a single range). Coalescing at this level would require copying the data but the caller may already have the needed data in a single buffer which would require no copying.

Thus, the implementation defers to the caller to coalesce subsequent requests if desired.

Fields

offset: u64

the virtual “offset” of this buffers (added to any request)

file_len: u64

The total length of the file being decoded

ranges: Vec<Range<u64>>

The ranges of data that are available for decoding (not adjusted for offset)

buffers: Vec<Bytes>

The buffers of data that can be used to decode the Parquet file

Implementations

impl PushBuffers

pub fn new(file_len: u64) -> Self

Create a new, empty PushBuffers for a file of the given length.

Use PushBuffers::default when the file length is unknown or irrelevant (e.g. the push decoder, which tracks ranges by absolute offset and never consults file_len).

pub fn push_ranges(&mut self, ranges: Vec<Range<u64>>, buffers: Vec<Bytes>)

Push all the ranges and buffers

pub fn push_range(&mut self, range: Range<u64>, buffer: Bytes)

Push a new range and its associated buffer

pub(crate) fn has_range(&self, range: &Range<u64>) -> bool

Returns true if the Buffers contains data for the given range

fn iter(&self) -> impl Iterator<Item = (&Range<u64>, &Bytes)>

pub(crate) fn file_len(&self) -> u64

return the file length of the Parquet file being read

fn with_offset(self, offset: u64) -> Self

Specify a new offset

pub(crate) fn buffered_bytes(&self) -> u64

Return the total of all buffered ranges

pub(crate) fn clear_ranges(&mut self, ranges_to_clear: &[Range<u64>])

Clear any range and corresponding buffer that is exactly in the ranges_to_clear

pub(crate) fn clear_all_ranges(&mut self)

Clear all buffered ranges and their corresponding data

Trait Implementations

impl ChunkReader for PushBuffers

type T = PushBuffers

The concrete type of reader returned by this trait

fn get_read(&self, start: u64) -> Result<Self::T, ParquetError>

Get a Read instance starting at the provided file offset

fn get_bytes(&self, start: u64, length: usize) -> Result<Bytes, ParquetError>

Get a range of data in memory as [Bytes]

impl Clone for PushBuffers

fn clone(&self) -> PushBuffers

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for PushBuffers

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Default for PushBuffers

fn default() -> PushBuffers

Returns the “default value” for a type.

impl Display for PushBuffers

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Length for PushBuffers

fn len(&self) -> u64

Returns the amount of bytes of the inner source.

impl Read for PushBuffers

less efficient implementation of Read for Buffers

fn read(&mut self, buf: &mut [u8]) -> Result<usize>

Pull some bytes from this source into the specified buffer, returning how many bytes were read.

fn read_vectored(&mut self, bufs: &mut [IoSliceMut<'_>]) -> Result<usize, Error>

Like read, except that it reads into a slice of buffers.

fn is_read_vectored(&self) -> bool

🔬This is a nightly-only experimental API. (can_vector)

Determines if this Reader has an efficient read_vectored implementation.

fn read_to_end(&mut self, buf: &mut Vec<u8>) -> Result<usize, Error>

Reads all bytes until EOF in this source, placing them into buf.

fn read_to_string(&mut self, buf: &mut String) -> Result<usize, Error>

Reads all bytes until EOF in this source, appending them to buf.

fn read_exact(&mut self, buf: &mut [u8]) -> Result<(), Error>

Reads the exact number of bytes required to fill buf.

fn read_buf(&mut self, buf: BorrowedCursor<'_>) -> Result<(), Error>

🔬This is a nightly-only experimental API. (read_buf)

Pull some bytes from this source into the specified buffer.

fn read_buf_exact(&mut self, cursor: BorrowedCursor<'_>) -> Result<(), Error>

🔬This is a nightly-only experimental API. (read_buf)

Reads the exact number of bytes required to fill cursor.

fn by_ref(&mut self) -> &mut Self

where Self: Sized,

Creates a “by reference” adapter for this instance of Read.

fn bytes(self) -> Bytes<Self>

where Self: Sized,

Transforms this Read instance to an Iterator over its bytes.

fn chain<R>(self, next: R) -> Chain<Self, R>

where R: Read, Self: Sized,

Creates an adapter which will chain this stream with another.

fn take(self, limit: u64) -> Take<Self>

where Self: Sized,

Creates an adapter which will read at most limit bytes from it.

fn read_array<const N: usize>(&mut self) -> Result<[u8; N], Error>

where Self: Sized,

🔬This is a nightly-only experimental API. (read_array)

Read and return a fixed array of bytes from this source.