Struct GenericRecordReader

pub struct GenericRecordReader<V, CV> {
    column_desc: ColumnDescPtr,
    values: V,
    def_levels: Option<DefinitionLevelBuffer>,
    rep_levels: Option<Vec<i16>>,
    column_reader: Option<GenericColumnReader<RepetitionLevelDecoderImpl, DefinitionLevelBufferDecoder, CV>>,
    num_values: usize,
    num_records: usize,
}

A generic stateful column reader that delimits semantic records

This type is hidden from the docs, and relies on private traits with no public implementations. As such this type signature may be changed without breaking downstream users as it can only be constructed through type aliases

Fields

column_desc: ColumnDescPtr

values: V

def_levels: Option<DefinitionLevelBuffer>

rep_levels: Option<Vec<i16>>

column_reader: Option<GenericColumnReader<RepetitionLevelDecoderImpl, DefinitionLevelBufferDecoder, CV>>

num_values: usize

Number of buffered levels / null-padded values

num_records: usize

Number of buffered records

Implementations

impl<V, CV> GenericRecordReader<V, CV>

where V: ValuesBuffer, CV: ColumnValueDecoder<Buffer = V>,

pub fn new(desc: ColumnDescPtr) -> Self

Create a new GenericRecordReader

pub fn set_page_reader( &mut self, page_reader: Box<dyn PageReader>, ) -> Result<()>

Set the current page reader.

pub fn read_records(&mut self, num_records: usize) -> Result<usize>

Try to read num_records of column data into internal buffer.

Returns

Number of actual records read.

pub fn skip_records(&mut self, num_records: usize) -> Result<usize>

Try to skip the next num_records rows

Returns

Number of records skipped

pub fn num_records(&self) -> usize

Returns number of records stored in buffer.

pub fn num_values(&self) -> usize

Return number of values stored in buffer. If the parquet column is not repeated, it should be equals to num_records, otherwise it should be larger than or equal to num_records.

pub fn consume_def_levels(&mut self) -> Option<Vec<i16>>

Returns definition level data. The implementation has side effects. It will create a new buffer to hold those definition level values that have already been read into memory but not counted as record values, e.g. those from self.num_values to self.values_written.

pub fn consume_rep_levels(&mut self) -> Option<Vec<i16>>

Return repetition level data. The side effect is similar to consume_def_levels.

pub fn consume_record_data(&mut self) -> V

Returns currently stored buffer data. The side effect is similar to consume_def_levels.

pub fn consume_bitmap_buffer(&mut self) -> Option<Buffer>

Returns currently stored null bitmap data for nullable columns. For non-nullable columns, the bitmap is discarded. The side effect is similar to consume_def_levels.

pub fn reset(&mut self)

Reset state of record reader. Should be called after consuming data, e.g. consume_rep_levels, consume_rep_levels, consume_record_data and consume_bitmap_buffer.

pub fn consume_bitmap(&mut self) -> Option<Buffer>

Returns bitmap data for nullable columns. For non-nullable columns, the bitmap is discarded.

fn read_one_batch(&mut self, batch_size: usize) -> Result<usize>

Try to read one batch of data returning the number of records read