Struct ArrowSbbf 

pub struct ArrowSbbf<'a> {
    sbbf: &'a Sbbf,
    arrow_type: DataType,
    physical_type: Type,
    type_length: usize,
}

A Parquet bloom filter (Sbbf) that can be checked with Arrow arrays

ArrowWriter may store an Arrow value as a different Parquet value, and the bloom filter holds the stored value, so Sbbf::check with the Arrow value can report a value that is present as absent. ArrowSbbf::check converts Arrow values in the same way as ArrowWriter before checking the filter.

let array: ArrayRef = Arc::new(Int8Array::from(vec![1, 2, 3]));
let batch = RecordBatch::try_from_iter([("c", array)]).unwrap();

let props = WriterProperties::builder()
    .set_bloom_filter_enabled(true)
    .build();
let mut buffer = Vec::new();
let mut writer = ArrowWriter::try_new(&mut buffer, batch.schema(), Some(props)).unwrap();
writer.write(&batch).unwrap();
writer.close().unwrap();

let options = ReadOptionsBuilder::new()
    .with_reader_properties(
        ReaderProperties::builder()
            .set_read_bloom_filter(true)
            .build(),
    )
    .build();
let reader = SerializedFileReader::new_with_options(Bytes::from(buffer), options).unwrap();
let row_group = reader.get_row_group(0).unwrap();
let sbbf = row_group.get_column_bloom_filter(0).unwrap();
let column = reader.metadata().row_group(0).column(0).column_descr();

// The Int8 values were written as INT32, so checking an i8 misses them
assert!(!sbbf.check(&1_i8));

// Use the column's Arrow type, here Int8, from the file's Arrow schema
let arrow_type = batch.schema().field(0).data_type().clone();
let sbbf = ArrowSbbf::try_new(sbbf, column, arrow_type).unwrap();
let found = sbbf.check(&Int8Array::from(vec![1, 3, 4])).unwrap();
assert_eq!(found, vec![Some(true), Some(true), Some(false)].into());

Fields

sbbf: &'a Sbbf

arrow_type: DataType

The Arrow type of the arrays to check

physical_type: Type

type_length: usize

The length of the column’s values if it is a FIXED_LEN_BYTE_ARRAY column, otherwise 0

Decimals are converted to this length, as other writers can store decimals in more bytes than their precision needs.

Implementations

impl<'a> ArrowSbbf<'a>

pub fn try_new( sbbf: &'a Sbbf, column: &ColumnDescriptor, arrow_type: DataType, ) -> Result<Self>

Create a new ArrowSbbf for the bloom filter sbbf of column

arrow_type is the Arrow type of the arrays that will be checked, for example the type of the column in the file’s Arrow schema.

Returns an error if values of arrow_type cannot be stored in column, if arrow_type is a decimal and column is not a decimal of the same scale, or the other way around, or if arrow_type is a time or timestamp in a different unit than column.

pub fn arrow_type(&self) -> &DataType

Returns the Arrow type of the arrays this ArrowSbbf checks

pub fn check(&self, array: &dyn Array) -> Result<BooleanArray>

Check each value of array against the bloom filter

Returns true if the value may be present, and false if it is definitely absent. Nulls in array are null in the result.

array must have the type returned by Self::arrow_type, or be a dictionary or run-end encoded array of values of that type, or a NullArray.

fn check_run_array<R: RunEndIndexType>( &self, array: &RunArray<R>, ) -> Result<BooleanArray>

Checks the values of a run-end encoded array, and repeats each result for its run

Trait Implementations

impl<'a> Clone for ArrowSbbf<'a>

fn clone(&self) -> Self

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl<'a> Debug for ArrowSbbf<'a>

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.