Struct GenericRecordReader 

pub struct GenericRecordReader<V, CV> {
    column_desc: ColumnDescPtr,
    values: Option<V>,
    def_levels: Option<DefinitionLevelBuffer>,
    rep_levels: Option<Vec<i16>>,
    column_reader: Option<GenericColumnReader<RepetitionLevelDecoderImpl, DefinitionLevelBufferDecoder, CV>>,
    num_values: usize,
    num_records: usize,
    capacity_hint: usize,
    values_written: usize,
    padding_threshold: Option<i16>,
    compact_bitmap: Option<BooleanBufferBuilder>,
}

A generic stateful column reader that delimits semantic records

This type is hidden from the docs, and relies on private traits with no public implementations. As such this type signature may be changed without breaking downstream users as it can only be constructed through type aliases

Fields

column_desc: ColumnDescPtr

values: Option<V>

Values buffer, lazily initialized on first read to avoid allocating a buffer that may never be used (e.g., after the last batch)

def_levels: Option<DefinitionLevelBuffer>

rep_levels: Option<Vec<i16>>

column_reader: Option<GenericColumnReader<RepetitionLevelDecoderImpl, DefinitionLevelBufferDecoder, CV>>

num_values: usize

Number of buffered levels / null-padded values

num_records: usize

Number of buffered records

capacity_hint: usize

Capacity hint for pre-allocating buffers based on batch size

values_written: usize

Number of values in the values buffer (may differ from num_values when padding_threshold is set, since parent-level padding is excluded).

padding_threshold: Option<i16>

Definition-level threshold used for selective null padding.

With full padding (None), the leaf values buffer has one slot for each decoded definition level. This includes placeholders for null or empty parent lists, which parent ListArrayReaders later have to filter out before computing offsets.

With selective padding (Some(threshold)), the threshold is the nearest enclosing list/map definition level. Entries with def < threshold describe a null/empty parent and are skipped entirely. Entries with def >= threshold belong to an actual child item slot: real values are copied, and item-level nulls are padded. The companion compact_bitmap has the same compact length and becomes the leaf null bitmap.

compact_bitmap: Option<BooleanBufferBuilder>

Compact bitmap accumulated during selective padding. Each bit corresponds to an item-level entry (def >= threshold): set when the value is real (def >= max_def), unset for item-level nulls. Used both as the valid_mask for pad_nulls (via as_slice()) and as the null bitmap consumed by the leaf reader (via consume_compact_bitmap).

Implementations

impl<V, CV> GenericRecordReader<V, CV>

where V: ValuesBuffer, CV: ColumnValueDecoder<Buffer = V>,

pub fn new(desc: ColumnDescPtr, capacity: usize) -> Self

Create a new GenericRecordReader

The capacity is used to pre-allocate internal buffers for full-padding reads, avoiding reallocations when reading fragmented row selections.

pub fn set_page_reader( &mut self, page_reader: Box<dyn PageReader>, ) -> Result<()>

Set the current page reader.

pub fn read_records(&mut self, num_records: usize) -> Result<usize>

Try to read num_records of column data into internal buffer.

Returns

Number of actual records read.

pub fn skip_records(&mut self, num_records: usize) -> Result<usize>

Try to skip the next num_records rows

Returns

Number of records skipped

pub fn num_records(&self) -> usize

Returns number of records stored in buffer.

pub fn num_values(&self) -> usize

Return number of values stored in buffer. If the parquet column is not repeated, it should be equals to num_records, otherwise it should be larger than or equal to num_records.

pub fn consume_def_levels(&mut self) -> Option<Vec<i16>>

Returns definition level data. The implementation has side effects. It will create a new buffer to hold those definition level values that have already been read into memory but not counted as record values, e.g. those from self.num_values to self.values_written.

pub fn consume_rep_levels(&mut self) -> Option<Vec<i16>>

Return repetition level data. The side effect is similar to consume_def_levels.

pub fn consume_record_data(&mut self) -> V

Returns currently stored buffer data. The side effect is similar to consume_def_levels.

pub fn reset(&mut self)

Reset state of record reader. Should be called after consuming data, e.g. consume_rep_levels, consume_rep_levels, consume_record_data and consume_compact_bitmap.

pub fn max_def_level(&self) -> i16

Returns the maximum definition level for the column being read.

pub fn set_padding_threshold(&mut self, threshold: i16)

Set the padding threshold. When set, pad_nulls only pads entries where def >= threshold (item-level nulls within non-null lists), skipping list-level padding entries (def < threshold).

pub fn values_written(&self) -> usize

Returns the number of values in the values buffer. When padding_threshold is None, this equals num_values (full padding). When padding_threshold is set, this is the item_count (selective padding).

pub fn consume_compact_bitmap(&mut self) -> Option<Buffer>

Consume the compact null bitmap built during selective padding. Returns the full bitmap when not using selective padding.

pub fn consume_bitmap(&mut self) -> Option<Buffer>

Returns bitmap data for nullable columns. For non-nullable columns, the bitmap is discarded.

fn read_one_batch(&mut self, batch_size: usize) -> Result<usize>

Try to read one batch of data returning the number of records read