parquet::arrow::arrow_reader::statistics

Macro make_decimal_stats_iterator

Source
macro_rules! make_decimal_stats_iterator {
    ($iterator_type:ident, $func:ident, $bytes_func:ident, $stat_value_type:ident, $convert_func: ident) => { ... };
}
Expand description

Special iterator adapter for extracting i128 values from from an iterator of ParquetStatistics

Handles checking if the statistics are present and valid with the correct type.

Depending on the parquet file, the statistics for Decimal128 can be stored as Int32, Int64 or ByteArray or FixedSizeByteArray :mindblown:

This iterator handles all cases, extracting the values and converting it to stat_value_type.

Parameters:

  • $iterator_type is the name of the iterator type (e.g. MinBooleanStatsIterator)
  • $func is the function to call to get the value (e.g. min or max)
  • $bytes_func is the function to call to get the value as bytes (e.g. min_bytes or max_bytes)
  • $stat_value_type is the type of the statistics value (e.g. i128)
  • convert_func is the function to convert the bytes to stats value (e.g. from_bytes_to_i128)