Struct SpawnedReader 

pub struct SpawnedReader<R> {
    inner: R,
    handle: Handle,
}

An AsyncFileReader that performs I/O on a separate tokio runtime.

Tokio is a cooperative scheduler, and relies on tasks yielding in a timely manner to service IO. Therefore, running IO and CPU-bound tasks, such as parquet decoding, on the same tokio runtime can lead to degraded throughput, dropped connections and other issues. For more information see here.

This wrapper spawns each operation of the inner reader onto the provided runtime [Handle], so that the runtime driving the parquet decoding does not also drive the I/O.

Note that Self::get_metadata spawns the entire metadata load, so the footer is also decoded on the provided runtime.

The inner reader must be Clone (typically an Arc’d handle to some shared resource) as each spawned task requires a 'static copy of it.

Fields

inner: R

handle: Handle

Implementations

impl<R> SpawnedReader<R>

pub fn new(inner: R, handle: Handle) -> Self

Creates a new SpawnedReader that performs the I/O of inner on handle

pub fn into_inner(self) -> R

Returns the inner reader

Trait Implementations

impl<R> AsyncFileReader for SpawnedReader<R>

where R: AsyncFileReader + Clone + Send + 'static,

fn get_bytes(&mut self, range: Range<u64>) -> BoxFuture<'_, Result<Bytes>>

Retrieve the bytes in range

fn get_byte_ranges( &mut self, ranges: Vec<Range<u64>>, ) -> BoxFuture<'_, Result<Vec<Bytes>>>

Retrieve multiple byte ranges. The default implementation will call get_bytes sequentially

fn get_metadata<'a>( &'a mut self, options: Option<&'a ArrowReaderOptions>, ) -> BoxFuture<'a, Result<Arc<ParquetMetaData>>>

Return a future which results in the ParquetMetaData for this Parquet file.

impl<R: Clone> Clone for SpawnedReader<R>

fn clone(&self) -> SpawnedReader<R>

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl<R: Debug> Debug for SpawnedReader<R>

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl<R> MetadataSuffixFetch for &mut SpawnedReader<R>

where R: AsyncFileReader + Clone + Send + 'static, for<'a> &'a mut R: MetadataSuffixFetch,

fn fetch_suffix(&mut self, suffix: usize) -> BoxFuture<'_, Result<Bytes>>

Return a future that fetches the last n bytes asynchronously