Struct AsyncArrowWriter

pub struct AsyncArrowWriter<W> {
    sync_writer: ArrowWriter<Vec<u8>>,
    async_writer: W,
}

Encodes [RecordBatch] to parquet, outputting to an AsyncFileWriter

Memory Usage

This writer eagerly writes data as soon as possible to the underlying AsyncFileWriter, permitting fine-grained control over buffering and I/O scheduling. However, the columnar nature of parquet forces data for an entire row group to be buffered in memory, before it can be flushed. Depending on the data and the configured row group size, this buffering may be substantial.

Memory usage can be limited by calling Self::flush to flush the in progress row group, although this will likely increase overall file size and reduce query performance. See ArrowWriter for more information.

let mut writer: AsyncArrowWriter<File> = todo!();
let batch: RecordBatch = todo!();
writer.write(&batch).await.unwrap();
// Trigger an early flush if buffered size exceeds 1_000_000
if writer.in_progress_size() > 1_000_000 {
    writer.flush().await.unwrap()
}

Fields

sync_writer: ArrowWriter<Vec<u8>>

Underlying sync writer

async_writer: W

Async writer provided by caller

Implementations

impl<W: AsyncFileWriter> AsyncArrowWriter<W>

pub fn try_new( writer: W, arrow_schema: SchemaRef, props: Option<WriterProperties>, ) -> Result<Self>

Try to create a new Async Arrow Writer

pub fn try_new_with_options( writer: W, arrow_schema: SchemaRef, options: ArrowWriterOptions, ) -> Result<Self>

Try to create a new Async Arrow Writer with ArrowWriterOptions

pub fn flushed_row_groups(&self) -> &[RowGroupMetaData]

Returns metadata for any flushed row groups

pub fn memory_size(&self) -> usize

Estimated memory usage, in bytes, of this ArrowWriter

See ArrowWriter::memory_size for more information.

pub fn in_progress_size(&self) -> usize

Anticipated encoded size of the in progress row group.

See ArrowWriter::memory_size for more information.

pub fn in_progress_rows(&self) -> usize

Returns the number of rows buffered in the in progress row group

pub fn bytes_written(&self) -> usize

Returns the number of bytes written by this instance

pub async fn write(&mut self, batch: &RecordBatch) -> Result<()>

Enqueues the provided RecordBatch to be written

After every sync write by the inner ArrowWriter, the inner buffer will be checked and flush if at least half full

pub async fn flush(&mut self) -> Result<()>

Flushes all buffered rows into a new row group

pub fn append_key_value_metadata(&mut self, kv_metadata: KeyValue)

Append KeyValue metadata in addition to those in WriterProperties

This method allows to append metadata after [RecordBatch]es are written.

pub async fn finish(&mut self) -> Result<FileMetaData>

Close and finalize the writer.

All the data in the inner buffer will be force flushed.

Unlike Self::close this does not consume self

Attempting to write after calling finish will result in an error

pub async fn close(self) -> Result<FileMetaData>

Close and finalize the writer.

All the data in the inner buffer will be force flushed.

async fn do_write(&mut self) -> Result<()>

Flush the data written by sync_writer into the async_writer

Notes

This method will take the inner buffer from the sync_writer and write it into the async writer. After the write, the inner buffer will be empty.