Module async_writer

Contains async writer which writes arrow data into parquet data.

Provides async API for writing [RecordBatch]es as parquet files. The API is similar to the sync API, so please read the documentation there before using this API.

Here is an example for using AsyncArrowWriter:

let col = Arc::new(Int64Array::from_iter_values([1, 2, 3])) as ArrayRef;
let to_write = RecordBatch::try_from_iter([("col", col)]).unwrap();

let mut buffer = Vec::new();
let mut writer = AsyncArrowWriter::try_new(&mut buffer, to_write.schema(), None).unwrap();
writer.write(&to_write).await.unwrap();
writer.close().await.unwrap();

let buffer = Bytes::from(buffer);
let mut reader = ParquetRecordBatchReaderBuilder::try_new(buffer.clone())
    .unwrap()
    .build()
    .unwrap();
let read = reader.next().unwrap().unwrap();

assert_eq!(to_write, read);

[object_store] provides it’s native implementation of AsyncFileWriter by ParquetObjectWriter.

Modules

• store 🔒

Structs

• AsyncArrowWriter
  Encodes [RecordBatch] to parquet, outputting to an AsyncFileWriter
• ParquetObjectWriter
  ParquetObjectWriter for writing to parquet to [ObjectStore]

Traits

• AsyncFileWriter
  The asynchronous interface used by AsyncArrowWriter to write parquet files.