Trait AsyncFileReader 

pub trait AsyncFileReader: Send {
    // Required methods
    fn get_bytes(&mut self, range: Range<u64>) -> BoxFuture<'_, Result<Bytes>>;
    fn get_metadata<'a>(
        &'a mut self,
        options: Option<&'a ArrowReaderOptions>,
    ) -> BoxFuture<'a, Result<Arc<ParquetMetaData>>>;

    // Provided method
    fn get_byte_ranges(
        &mut self,
        ranges: Vec<Range<u64>>,
    ) -> BoxFuture<'_, Result<Vec<Bytes>>> { ... }
}

The asynchronous interface used by ParquetRecordBatchStream to read parquet files

Notes:

1. There is a default implementation for types that implement [AsyncRead] and [AsyncSeek], for example tokio::fs::File.

2. Implementations for remote storage, such as the object_store crate, can implement this interface directly, typically by pairing a store handle with an object path and delegating Self::get_bytes and Self::get_byte_ranges to ranged reads. SpawnedReader can wrap such a reader to perform its I/O on a dedicated runtime, and ParquetMetaDataReader::with_arrow_reader_options simplifies implementing Self::get_metadata.

Example: implementing AsyncFileReader for the object_store crate

use bytes::Bytes;
use futures::future::BoxFuture;
use futures::{FutureExt, TryFutureExt};
use object_store::path::Path;
use object_store::{GetOptions, GetRange, ObjectStore, ObjectStoreExt};
use parquet::arrow::arrow_reader::ArrowReaderOptions;
use parquet::arrow::async_reader::{AsyncFileReader, MetadataSuffixFetch};
use parquet::errors::{ParquetError, Result};
use parquet::file::metadata::{ParquetMetaData, ParquetMetaDataReader};

fn to_parquet_err(e: object_store::Error) -> ParquetError {
    ParquetError::External(Box::new(e))
}

#[derive(Clone)]
struct ObjectStoreReader {
    store: Arc<dyn ObjectStore>,
    path: Path,
}

impl AsyncFileReader for ObjectStoreReader {
    fn get_bytes(&mut self, range: Range<u64>) -> BoxFuture<'_, Result<Bytes>> {
        self.store
            .get_range(&self.path, range)
            .map_err(to_parquet_err)
            .boxed()
    }

    fn get_byte_ranges(&mut self, ranges: Vec<Range<u64>>) -> BoxFuture<'_, Result<Vec<Bytes>>> {
        async move {
            self.store
                .get_ranges(&self.path, &ranges)
                .await
                .map_err(to_parquet_err)
        }
        .boxed()
    }

    fn get_metadata<'a>(
        &'a mut self,
        options: Option<&'a ArrowReaderOptions>,
    ) -> BoxFuture<'a, Result<Arc<ParquetMetaData>>> {
        async move {
            let metadata = ParquetMetaDataReader::new()
                .with_arrow_reader_options(options)
                .load_via_suffix_and_finish(self)
                .await?;
            Ok(Arc::new(metadata))
        }
        .boxed()
    }
}

/// Supports fetching the parquet footer without knowing the file size,
/// via suffix range requests
impl MetadataSuffixFetch for &mut ObjectStoreReader {
    fn fetch_suffix(&mut self, suffix: usize) -> BoxFuture<'_, Result<Bytes>> {
        let options = GetOptions {
            range: Some(GetRange::Suffix(suffix as u64)),
            ..Default::default()
        };
        async move {
            let resp = self
                .store
                .get_opts(&self.path, options)
                .await
                .map_err(to_parquet_err)?;
            resp.bytes().await.map_err(to_parquet_err)
        }
        .boxed()
    }
}

Required Methods

fn get_bytes(&mut self, range: Range<u64>) -> BoxFuture<'_, Result<Bytes>>

Retrieve the bytes in range

fn get_metadata<'a>( &'a mut self, options: Option<&'a ArrowReaderOptions>, ) -> BoxFuture<'a, Result<Arc<ParquetMetaData>>>

Return a future which results in the ParquetMetaData for this Parquet file.

This is an asynchronous operation as it may involve reading the file footer and potentially other metadata from disk or a remote source.

Reading data from Parquet requires the metadata to understand the schema, row groups, and location of pages within the file. This metadata is stored primarily in the footer of the Parquet file, and can be read using ParquetMetaDataReader.

However, implementations can significantly speed up reading Parquet by supplying cached metadata or pre-fetched metadata via this API.

Parameters

• options: Optional ArrowReaderOptions that may contain decryption and other options that affect how the metadata is read.

Provided Methods

fn get_byte_ranges( &mut self, ranges: Vec<Range<u64>>, ) -> BoxFuture<'_, Result<Vec<Bytes>>>

Retrieve multiple byte ranges. The default implementation will call get_bytes sequentially

Trait Implementations

impl AsyncFileReader for Box<dyn AsyncFileReader + '_>

This allows Box<dyn AsyncFileReader + ’_> to be used as an AsyncFileReader,

fn get_bytes(&mut self, range: Range<u64>) -> BoxFuture<'_, Result<Bytes>>

Retrieve the bytes in range

fn get_byte_ranges( &mut self, ranges: Vec<Range<u64>>, ) -> BoxFuture<'_, Result<Vec<Bytes>>>

Retrieve multiple byte ranges. The default implementation will call get_bytes sequentially

fn get_metadata<'a>( &'a mut self, options: Option<&'a ArrowReaderOptions>, ) -> BoxFuture<'a, Result<Arc<ParquetMetaData>>>

Return a future which results in the ParquetMetaData for this Parquet file.

Dyn Compatibility

This trait is dyn compatible.

In older versions of Rust, dyn compatibility was called "object safety".

Implementations on Foreign Types

impl AsyncFileReader for Box<dyn AsyncFileReader + '_>

This allows Box<dyn AsyncFileReader + ’_> to be used as an AsyncFileReader,

fn get_bytes(&mut self, range: Range<u64>) -> BoxFuture<'_, Result<Bytes>>

fn get_byte_ranges( &mut self, ranges: Vec<Range<u64>>, ) -> BoxFuture<'_, Result<Vec<Bytes>>>

fn get_metadata<'a>( &'a mut self, options: Option<&'a ArrowReaderOptions>, ) -> BoxFuture<'a, Result<Arc<ParquetMetaData>>>

Implementors

impl AsyncFileReader for ParquetObjectReader

impl<R> AsyncFileReader for SpawnedReader<R>

where R: AsyncFileReader + Clone + Send + 'static,

impl<T: AsyncRead + AsyncSeek + Unpin + Send> AsyncFileReader for T