Module async_writer 

async API for writing [RecordBatch]es to Parquet files

See the crate-level documentation for more details.

The async API for writing [RecordBatch]es is similar to the sync API, so please read the documentation there before using this API.

Here is an example for using AsyncArrowWriter:

let col = Arc::new(Int64Array::from_iter_values([1, 2, 3])) as ArrayRef;
let to_write = RecordBatch::try_from_iter([("col", col)]).unwrap();

let mut buffer = Vec::new();
let mut writer = AsyncArrowWriter::try_new(&mut buffer, to_write.schema(), None).unwrap();
writer.write(&to_write).await.unwrap();
writer.close().await.unwrap();

let buffer = Bytes::from(buffer);
let mut reader = ParquetRecordBatchReaderBuilder::try_new(buffer.clone())
    .unwrap()
    .build()
    .unwrap();
let read = reader.next().unwrap().unwrap();

assert_eq!(to_write, read);

There is a blanket implementation of AsyncFileWriter for all types that implement [AsyncWrite], so writers such as tokio::fs::File, or object_store::buffered::BufWriter for writing to object storage, can be passed to AsyncArrowWriter directly.

Re-exports

use crate::arrow::ArrowWriter; 🔒

use crate::arrow::arrow_writer::ArrowWriterOptions; 🔒

use crate::errors::ParquetError; 🔒

use crate::errors::Result; 🔒

use crate::file::metadata::KeyValue; 🔒

use crate::file::metadata::ParquetMetaData; 🔒

use crate::file::metadata::RowGroupMetaData; 🔒

use crate::file::properties::WriterProperties; 🔒

use arrow_array::RecordBatch; 🔒

use arrow_schema::SchemaRef; 🔒

use bytes::Bytes; 🔒

use futures::FutureExt; 🔒

use futures::future::BoxFuture; 🔒

use std::mem; 🔒

use tokio::io::AsyncWrite; 🔒

use tokio::io::AsyncWriteExt; 🔒

Modules

store 🔒

Structs

AsyncArrowWriter

Encodes [RecordBatch] to parquet, outputting to an AsyncFileWriter

ParquetObjectWriterDeprecated

ParquetObjectWriter for writing to parquet to [ObjectStore]

Traits

AsyncFileWriter

The asynchronous interface used by AsyncArrowWriter to write parquet files.