Struct SerializedRowGroupWriter

pub struct SerializedRowGroupWriter<'a, W: Write> {

    descr: SchemaDescPtr,
    props: WriterPropertiesPtr,
    buf: &'a mut TrackedWrite<W>,
    total_rows_written: Option<u64>,
    total_bytes_written: u64,
    total_uncompressed_bytes: i64,
    column_index: usize,
    row_group_metadata: Option<RowGroupMetaDataPtr>,
    column_chunks: Vec<ColumnChunkMetaData>,
    bloom_filters: Vec<Option<Sbbf>>,
    column_indexes: Vec<Option<ColumnIndex>>,
    offset_indexes: Vec<Option<OffsetIndex>>,
    row_group_index: i16,
    file_offset: i64,
    on_close: Option<OnCloseRowGroup<'a, W>>,
}

Parquet row group writer API. Provides methods to access column writers in an iterator-like fashion, order is guaranteed to match the order of schema leaves (column descriptors).

All columns should be written sequentially; the main workflow is:

• Request the next column using next_column method - this will return None if no more columns are available to write.
• Once done writing a column, close column writer with close
• Once all columns have been written, close row group writer with close method. THe close method will return row group metadata and is no-op on already closed row group.

Fields

descr: SchemaDescPtr

props: WriterPropertiesPtr

buf: &'a mut TrackedWrite<W>

total_rows_written: Option<u64>

total_bytes_written: u64

total_uncompressed_bytes: i64

column_index: usize

row_group_metadata: Option<RowGroupMetaDataPtr>

column_chunks: Vec<ColumnChunkMetaData>

bloom_filters: Vec<Option<Sbbf>>

column_indexes: Vec<Option<ColumnIndex>>

offset_indexes: Vec<Option<OffsetIndex>>

row_group_index: i16

file_offset: i64

on_close: Option<OnCloseRowGroup<'a, W>>

Implementations

impl<'a, W: Write + Send> SerializedRowGroupWriter<'a, W>

pub fn new( schema_descr: SchemaDescPtr, properties: WriterPropertiesPtr, buf: &'a mut TrackedWrite<W>, row_group_index: i16, on_close: Option<OnCloseRowGroup<'a, W>>, ) -> Self

Creates a new SerializedRowGroupWriter with:

• schema_descr - the schema to write
• properties - writer properties
• buf - the buffer to write data to
• row_group_index - row group index in this parquet file.
• file_offset - file offset of this row group in this parquet file.
• on_close - an optional callback that will invoked on Self::close

fn next_column_desc(&mut self) -> Option<ColumnDescPtr>

Advance self.column_index returning the next ColumnDescPtr if any

fn get_on_close(&mut self) -> (&mut TrackedWrite<W>, OnCloseColumnChunk<'_>)

Returns OnCloseColumnChunk for the next writer

pub(crate) fn next_column_with_factory<'b, F, C>( &'b mut self, factory: F, ) -> Result<Option<C>>

where F: FnOnce(ColumnDescPtr, WriterPropertiesPtr, Box<dyn PageWriter + 'b>, OnCloseColumnChunk<'b>) -> Result<C>,

Returns the next column writer, if available, using the factory function; otherwise returns None.

pub fn next_column(&mut self) -> Result<Option<SerializedColumnWriter<'_>>>

Returns the next column writer, if available; otherwise returns None. In case of any IO error or Thrift error, or if row group writer has already been closed returns Err.

pub fn append_column<R: ChunkReader>( &mut self, reader: &R, close: ColumnCloseResult, ) -> Result<()>

Append an encoded column chunk from another source without decoding it

This can be used for efficiently concatenating or projecting parquet data, or encoding parquet data to temporary in-memory buffers

See Self::next_column for writing data that isn’t already encoded

pub fn close(self) -> Result<RowGroupMetaDataPtr>

Closes this row group writer and returns row group metadata.

fn assert_previous_writer_closed(&self) -> Result<()>