Struct StreamingColumnChunkReader 

struct StreamingColumnChunkReader {
    dictionary: IntoIter<Bytes>,
    store: Box<dyn PageStore>,
    keys: IntoIter<PageKey>,
    current: Bytes,
}

A streaming Read over one column chunk’s buffered pages, in final file order: the in-memory dictionary page (if any) first, then the data pages.

Each data-page blob is taken back out of the PageStore as it is consumed and released immediately afterwards, so splicing a chunk into the output file never materializes more than a single page in memory at a time. This is what keeps the splice phase within the memory bound for a spilling backend (an in-memory store already holds the bytes, so it is unaffected).

Fields

dictionary: IntoIter<Bytes>

Dictionary-page blobs, emitted before any data page.

store: Box<dyn PageStore>

keys: IntoIter<PageKey>

current: Bytes

The blob currently being drained into the output; emptied as it is read.

Implementations

impl StreamingColumnChunkReader

fn new(data: ArrowColumnChunkData) -> Self

Trait Implementations

impl Read for StreamingColumnChunkReader

fn read(&mut self, out: &mut [u8]) -> Result<usize>

Pull some bytes from this source into the specified buffer, returning how many bytes were read.

fn read_vectored(&mut self, bufs: &mut [IoSliceMut<'_>]) -> Result<usize, Error>

Like read, except that it reads into a slice of buffers.

fn is_read_vectored(&self) -> bool

🔬This is a nightly-only experimental API. (can_vector)

Determines if this Reader has an efficient read_vectored implementation.

fn read_to_end(&mut self, buf: &mut Vec<u8>) -> Result<usize, Error>

Reads all bytes until EOF in this source, placing them into buf.

fn read_to_string(&mut self, buf: &mut String) -> Result<usize, Error>

Reads all bytes until EOF in this source, appending them to buf.

fn read_exact(&mut self, buf: &mut [u8]) -> Result<(), Error>

Reads the exact number of bytes required to fill buf.

fn read_buf(&mut self, buf: BorrowedCursor<'_>) -> Result<(), Error>

🔬This is a nightly-only experimental API. (read_buf)

Pull some bytes from this source into the specified buffer.

fn read_buf_exact(&mut self, cursor: BorrowedCursor<'_>) -> Result<(), Error>

🔬This is a nightly-only experimental API. (read_buf)

Reads the exact number of bytes required to fill cursor.

fn by_ref(&mut self) -> &mut Self

where Self: Sized,

Creates a “by reference” adapter for this instance of Read.

fn bytes(self) -> Bytes<Self>

where Self: Sized,

Transforms this Read instance to an Iterator over its bytes.

fn chain<R>(self, next: R) -> Chain<Self, R>

where R: Read, Self: Sized,

Creates an adapter which will chain this stream with another.

fn take(self, limit: u64) -> Take<Self>

where Self: Sized,

Creates an adapter which will read at most limit bytes from it.

fn read_array<const N: usize>(&mut self) -> Result<[u8; N], Error>

where Self: Sized,

🔬This is a nightly-only experimental API. (read_array)

Read and return a fixed array of bytes from this source.