parquet::file

Module metadata

Source
Expand description

Parquet metadata API

Most users should use these structures to interact with Parquet metadata. The crate::format module contains lower level structures generated from the Parquet thrift definition.

  • ParquetMetaData: Top level metadata container, read from the Parquet file footer.

  • FileMetaData: File level metadata such as schema, row counts and version.

  • RowGroupMetaData: Metadata for each Row Group with a File, such as location and number of rows, and column chunks.

  • ColumnChunkMetaData: Metadata for each column chunk (primitive leaf) within a Row Group including encoding and compression information, number of values, statistics, etc.

§APIs for working with Parquet Metadata

The Parquet readers and writers in this crate handle reading and writing metadata into parquet files. To work with metadata directly, the following APIs are available:

§Examples

Please see external_metadata.rs

§Metadata Encodings and Structures

There are three different encodings of Parquet Metadata in this crate:

  1. bytes:encoded with the Thrift TCompactProtocol as defined in parquet.thrift

  2. format: Rust structures automatically generated by the thrift compiler from parquet.thrift. These structures are low level and mirror the thrift definitions.

  3. file::metadata (this module): Easier to use Rust structures with a more idiomatic API. Note that, confusingly, some but not all of these structures have the same name as the format structures.

Graphically, this is how the different structures relate to each other:

                         ┌ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─         ┌ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─
                           ┌──────────────┐     │         ┌───────────────────────┐ │
                         │ │ ColumnIndex  │              ││    ParquetMetaData    │
                           └──────────────┘     │         └───────────────────────┘ │
┌──────────────┐         │ ┌────────────────┐            │┌───────────────────────┐
│   ..0x24..   │ ◀────▶    │  OffsetIndex   │   │ ◀────▶  │    ParquetMetaData    │ │
└──────────────┘         │ └────────────────┘            │└───────────────────────┘
                                    ...         │                   ...             │
                         │ ┌──────────────────┐          │ ┌──────────────────┐
bytes                      │  FileMetaData*   │ │          │  FileMetaData*   │     │
(thrift encoded)         │ └──────────────────┘          │ └──────────────────┘
                          ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ┘         ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ┘

                         format::meta structures          file::metadata structures

                        * Same name, different struct

Modules§

memory 🔒
Memory calculations for ParquetMetadata::memory_size
reader 🔒
writer 🔒

Structs§

ColumnChunkMetaData
Metadata for a column chunk.
ColumnChunkMetaDataBuilder
Builder for ColumnChunkMetaData
ColumnIndexBuilder
Builder for Parquet ColumnIndex, part of the Parquet PageIndex
FileMetaData
File level metadata for a Parquet file.
LevelHistogram
Histograms for repetition and definition levels.
OffsetIndexBuilder
Builder for offset index, part of the Parquet PageIndex.
ParquetMetaData
Parsed metadata for a single Parquet file
ParquetMetaDataBuilder
A builder for creating / manipulating ParquetMetaData
ParquetMetaDataReader
Reads the ParquetMetaData from a byte stream.
ParquetMetaDataWriter
Writes ParquetMetaData to a byte stream
RowGroupMetaData
Metadata for a row group
RowGroupMetaDataBuilder
Builder for row group metadata.

Type Aliases§

FileMetaDataPtr
Reference counted pointer for FileMetaData.
KeyValue
A key-value pair for FileMetaData.
ParquetColumnIndex
Page level statistics for each column chunk of each row group.
ParquetOffsetIndex
OffsetIndexMetaData for each data page of each row group of each column
RowGroupMetaDataPtr
Reference counted pointer for RowGroupMetaData.