Module metadata

parquet::file

Module metadata 

Source
Expand description

Parquet metadata API

Users should use these structures to interact with Parquet metadata.

  • ParquetMetaData: Top level metadata container, read from the Parquet file footer.

  • FileMetaData: File level metadata such as schema, row counts and version.

  • RowGroupMetaData: Metadata for each Row Group with a File, such as location and number of rows, and column chunks.

  • ColumnChunkMetaData: Metadata for each column chunk (primitive leaf) within a Row Group including encoding and compression information, number of values, statistics, etc.

§APIs for working with Parquet Metadata

The Parquet readers and writers in this crate handle reading and writing metadata into parquet files. To work with metadata directly, the following APIs are available:

§Examples

Please see external_metadata.rs

§Metadata Encodings and Structures

There are three different encodings of Parquet Metadata in this crate:

  1. bytes:encoded with the Thrift TCompactProtocol as defined in parquet.thrift

  2. format: Rust structures automatically generated by the thrift compiler from parquet.thrift. These structures are low level and mirror the thrift definitions.

  3. file::metadata (this module): Easier to use Rust structures with a more idiomatic API. Note that, confusingly, some but not all of these structures have the same name as the format structures.

Graphically, this is how the different structures relate to each other:

                         ┌ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─         ┌ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─
                           ┌──────────────┐     │         ┌───────────────────────┐ │
                         │ │ ColumnIndex  │              ││    ParquetMetaData    │
                           └──────────────┘     │         └───────────────────────┘ │
┌──────────────┐         │ ┌────────────────┐            │┌───────────────────────┐
│   ..0x24..   │ ◀────▶    │  OffsetIndex   │   │ ◀────▶  │    ParquetMetaData    │ │
└──────────────┘         │ └────────────────┘            │└───────────────────────┘
                                    ...         │                   ...             │
                         │ ┌──────────────────┐          │ ┌──────────────────┐
bytes                      │  FileMetaData*   │ │          │  FileMetaData*   │     │
(thrift encoded)         │ └──────────────────┘          │ └──────────────────┘
                          ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ┘         ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ┘

                         format::meta structures          file::metadata structures

                        * Same name, different struct

Modules§

footer_tail 🔒
memory 🔒
Memory calculations for ParquetMetadata::memory_size
parser 🔒
Internal metadata parsing routines
push_decoder 🔒
reader 🔒
thrift 🔒
This module is the bridge between a Parquet file’s thrift encoded metadata and this crate’s Parquet metadata API. It contains objects and functions used to serialize/deserialize metadata objects into/from the Thrift compact protocol format as defined by the Parquet specification.
writer 🔒

Structs§

ColumnChunkMetaData
Metadata for a column chunk.
ColumnChunkMetaDataBuilder
Builder for ColumnChunkMetaData
ColumnIndexBuilder
Builder for Parquet ColumnIndex, part of the Parquet PageIndex
FileMetaData
File level metadata for a Parquet file.
FooterTail
Parsed Parquet footer tail (last 8 bytes of a Parquet file)
KeyValue
A key-value pair for FileMetaData.
LevelHistogram
Histograms for repetition and definition levels.
OffsetIndexBuilder
Builder for offset index, part of the Parquet PageIndex.
PageEncodingStats
PageEncodingStats for a column chunk and data page.
ParquetMetaData
Parsed metadata for a single Parquet file
ParquetMetaDataBuilder
A builder for creating / manipulating ParquetMetaData
ParquetMetaDataPushDecoder
A push decoder for ParquetMetaData.
ParquetMetaDataReader
Reads ParquetMetaData from a byte stream, with either synchronous or asynchronous I/O.
ParquetMetaDataWriter
Writes ParquetMetaData to a byte stream
RowGroupMetaData
Metadata for a row group
RowGroupMetaDataBuilder
Builder for row group metadata.
SortingColumn
Sort order within a RowGroup of a leaf column

Enums§

PageIndexPolicy
Describes the policy for reading page indexes

Type Aliases§

FileMetaDataPtr
Reference counted pointer for FileMetaData.
ParquetColumnIndex
Page level statistics for each column chunk of each row group.
ParquetOffsetIndex
OffsetIndexMetaData for each data page of each row group of each column
RowGroupMetaDataPtr
Reference counted pointer for RowGroupMetaData.