Module async_writer 

async API for writing [RecordBatch]es to Parquet files

See the crate-level documentation for more details.

The async API for writing [RecordBatch]es is similar to the sync API, so please read the documentation there before using this API.

Here is an example for using AsyncArrowWriter:

let col = Arc::new(Int64Array::from_iter_values([1, 2, 3])) as ArrayRef;
let to_write = RecordBatch::try_from_iter([("col", col)]).unwrap();

let mut buffer = Vec::new();
let mut writer = AsyncArrowWriter::try_new(&mut buffer, to_write.schema(), None).unwrap();
writer.write(&to_write).await.unwrap();
writer.close().await.unwrap();

let buffer = Bytes::from(buffer);
let mut reader = ParquetRecordBatchReaderBuilder::try_new(buffer.clone())
    .unwrap()
    .build()
    .unwrap();
let read = reader.next().unwrap().unwrap();

assert_eq!(to_write, read);

[object_store] provides it’s native implementation of AsyncFileWriter by ParquetObjectWriter.

Modules

store 🔒

Structs

AsyncArrowWriter

Encodes [RecordBatch] to parquet, outputting to an AsyncFileWriter

ParquetObjectWriter

ParquetObjectWriter for writing to parquet to [ObjectStore]

Traits

AsyncFileWriter

The asynchronous interface used by AsyncArrowWriter to write parquet files.