Struct PushBuffers 

pub struct PushBuffers {
    offset: u64,
    file_len: u64,
    ranges: Vec<Range<u64>>,
    buffers: Vec<Bytes>,
    max_len: u64,
    buffered_bytes: u64,
}

Holds multiple non-contiguous, caller-provided buffers of file data.

This is the in-memory buffer used by the push-based Parquet decoders (ParquetPushDecoder and ParquetMetaDataPushDecoder). It can be constructed up front and handed to a builder so the decoder reuses bytes that have already been fetched.

Features:

1. Zero copy
2. non contiguous ranges of bytes

Non Coalescing

This buffer does not coalesce (merging adjacent ranges of bytes into a single range). Coalescing at this level would require copying the data but the caller may already have the needed data in a single buffer which would require no copying.

Thus, the implementation defers to the caller to coalesce subsequent requests if desired.

Ordering

The buffers are sorted by the start of their range. Thus, lookups use a binary search, not a scan of all buffers. Pushed ranges can overlap.

Fields

offset: u64

the virtual “offset” of this buffers (added to any request)

file_len: u64

The total length of the file being decoded

ranges: Vec<Range<u64>>

The ranges of data that are available for decoding (not adjusted for offset), sorted by start

buffers: Vec<Bytes>

The buffers of data that can be used to decode the Parquet file, in the same order as ranges

max_len: u64

The length of the longest range in ranges.

This keeps lookups fast in the common case: buffers that do not overlap and have similar lengths. Pushed ranges can overlap, so a lookup cannot stop at the nearest buffer. max_len tells the lookup when no earlier buffer can reach the requested range, so it checks one or two buffers, not all of them. See Self::find.

buffered_bytes: u64

The sum of the lengths of ranges, kept up to date so that Self::buffered_bytes does not scan all ranges.

Implementations

impl PushBuffers

pub fn new(file_len: u64) -> Self

Create a new, empty PushBuffers for a file of the given length.

Use PushBuffers::default when the file length is unknown or irrelevant (e.g. the push decoder, which tracks ranges by absolute offset and never consults file_len).

pub fn push_ranges( &mut self, ranges: Vec<Range<u64>>, buffers: Vec<Bytes>, ) -> Result<(), ParquetError>

Push all the ranges and buffers

Errors

Returns an error if the number of ranges does not match the number of buffers, or if any buffer’s length does not match its range (see Self::push_range).

pub fn push_range( &mut self, range: Range<u64>, buffer: Bytes, ) -> Result<(), ParquetError>

Push a new range and its associated buffer

Errors

Returns an error if the buffer’s length does not match the range’s length, e.g. when a truncated (short) read is pushed.

pub(crate) fn has_range(&self, range: &Range<u64>) -> bool

Returns true if the Buffers contains data for the given range

fn find(&self, start: u64, end: u64) -> Option<usize>

Returns the index of a buffer that contains all bytes of start..end, if any.

pub(crate) fn file_len(&self) -> u64

return the file length of the Parquet file being read

fn with_offset(self, offset: u64) -> Self

Specify a new offset

pub(crate) fn buffered_bytes(&self) -> u64

Return the total length of all buffered ranges.

This counts the bytes that the decoder can still read, not the allocated memory. Self::release_ranges keeps the remaining parts of a buffer as slices of the pushed [Bytes], and the allocation of a pushed [Bytes] is freed only after all of its parts are released.

pub(crate) fn clear_ranges(&mut self, ranges_to_clear: &[Range<u64>])

Clear any range and corresponding buffer that is exactly in the ranges_to_clear

fn update_max_len(&mut self)

Set max_len to the maximum length of the remaining ranges.

A max_len that is too large is still correct, lookups only scan further. This update is for performance: a large buffer that was removed must not slow down later lookups.

pub(crate) fn release_ranges(&mut self, ranges: &[Range<u64>])

Remove all buffered bytes in ranges.

A buffer that overlaps a range is trimmed or split, and the parts outside the range are kept. The kept parts are zero-copy slices, which record the bytes that the decoder can still read. Thus, the allocator frees the memory of a pushed [Bytes] only after all of its parts are removed.

The buffers stay sorted by start, which lookups require (see Self::find). If no buffer overlaps ranges, the buffers do not change.

pub(crate) fn retain_ranges(&mut self, keep: &[Range<u64>])

Remove all buffered bytes outside keep.

This calls Self::release_ranges with the complement of keep, so the same rules apply to the kept parts.

pub(crate) fn clear_all_ranges(&mut self)

Clear all buffered ranges and their corresponding data

Trait Implementations

impl ChunkReader for PushBuffers

type T = PushBuffers

The concrete type of reader returned by this trait

fn get_read(&self, start: u64) -> Result<Self::T, ParquetError>

Get a Read instance starting at the provided file offset

fn get_bytes(&self, start: u64, length: usize) -> Result<Bytes, ParquetError>

Get a range of data in memory as [Bytes]

impl Clone for PushBuffers

fn clone(&self) -> Self

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for PushBuffers

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Default for PushBuffers

fn default() -> Self

Returns the “default value” for a type.

impl Display for PushBuffers

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Length for PushBuffers

fn len(&self) -> u64

Returns the amount of bytes of the inner source.

impl Read for PushBuffers

less efficient implementation of Read for Buffers

fn read(&mut self, buf: &mut [u8]) -> Result<usize>

Pull some bytes from this source into the specified buffer, returning how many bytes were read.

fn read_vectored(&mut self, bufs: &mut [IoSliceMut<'_>]) -> Result<usize, Error>

Like read, except that it reads into a slice of buffers.

fn is_read_vectored(&self) -> bool

🔬This is a nightly-only experimental API. (can_vector)

Determines if this Reader has an efficient read_vectored implementation.

fn read_to_end(&mut self, buf: &mut Vec<u8>) -> Result<usize, Error>

Reads all bytes until EOF in this source, placing them into buf.

fn read_to_string(&mut self, buf: &mut String) -> Result<usize, Error>

Reads all bytes until EOF in this source, appending them to buf.

fn read_exact(&mut self, buf: &mut [u8]) -> Result<(), Error>

Reads the exact number of bytes required to fill buf.

fn read_buf(&mut self, buf: BorrowedCursor<'_, u8>) -> Result<(), Error>

🔬This is a nightly-only experimental API. (read_buf)

Pull some bytes from this source into the specified buffer.

fn read_buf_exact( &mut self, cursor: BorrowedCursor<'_, u8>, ) -> Result<(), Error>

🔬This is a nightly-only experimental API. (read_buf)

Reads the exact number of bytes required to fill cursor.

fn by_ref(&mut self) -> &mut Self

where Self: Sized,

Creates a “by reference” adapter for this instance of Read.

fn bytes(self) -> Bytes<Self>

where Self: Sized,

Transforms this Read instance to an Iterator over its bytes.

fn chain<R>(self, next: R) -> Chain<Self, R>

where R: Read, Self: Sized,

Creates an adapter which will chain this stream with another.

fn take(self, limit: u64) -> Take<Self>

where Self: Sized,

Creates an adapter which will read at most limit bytes from it.

fn read_array<const N: usize>(&mut self) -> Result<[u8; N], Error>

where Self: Sized,

🔬This is a nightly-only experimental API. (read_array)

Read and return a fixed array of bytes from this source.

fn read_le<T>(&mut self) -> Result<T, Error>

where T: FromEndianBytes, Self: Sized,

🔬This is a nightly-only experimental API. (read_le)

Read and return a type (e.g. an integer) in little-endian order.

fn read_be<T>(&mut self) -> Result<T, Error>

where T: FromEndianBytes, Self: Sized,

🔬This is a nightly-only experimental API. (read_le)

Read and return a type (e.g. an integer) in big-endian order.