parquet::arrow

Function parquet_to_arrow_field_levels

Source
pub fn parquet_to_arrow_field_levels(
    schema: &SchemaDescriptor,
    mask: ProjectionMask,
    hint: Option<&Fields>,
) -> Result<FieldLevels>
Expand description

Convert a parquet SchemaDescriptor to FieldLevels

Columns not included within ProjectionMask will be ignored.

Where a field type in hint is compatible with the corresponding parquet type in schema, it will be used, otherwise the default arrow type for the given parquet column type will be used.

This is to accommodate arrow types that cannot be round-tripped through parquet natively. Depending on the parquet writer, this can lead to a mismatch between a file’s parquet schema and its embedded arrow schema. The parquet schema must be treated as authoritative in such an event. See #1663 for more information

Note: this is a low-level API, most users will want to make use of the higher-level parquet_to_arrow_schema for decoding metadata from a parquet file.