Module metadata 

Parquet metadata API

Users should use these structures to interact with Parquet metadata.

• ParquetMetaData: Top level metadata container, read from the Parquet file footer.

• FileMetaData: File level metadata such as schema, row counts and version.

• RowGroupMetaData: Metadata for each Row Group with a File, such as location and number of rows, and column chunks.

• ColumnChunkMetaData: Metadata for each column chunk (primitive leaf) within a Row Group including encoding and compression information, number of values, statistics, etc.

• PageIndex: Metadata and statistics used to reduce page-level I/O.

APIs for working with Parquet Metadata

The Parquet readers and writers in this crate handle reading and writing metadata into Parquet files. To work with metadata directly, the following APIs are available:

• ParquetMetaDataReader for reading metadata from an I/O source (sync and async)
• ParquetMetaDataPushDecoder for decoding from bytes without I/O
• ParquetMetaDataWriter for writing.

Examples

Please see external_metadata.rs

Modules

dictionary 🔒

Decoding a column chunk’s dictionary page directly into an Arrow array, independent of the row-by-row ArrayReader machinery.

footer_tail 🔒

memory 🔒

Memory calculations for ParquetMetadata::memory_size

options 🔒

Options used to control metadata parsing

page_index

Page Index structures for efficient page-level skipping

parser 🔒

Internal metadata parsing routines

push_decoder 🔒

reader 🔒

thrift 🔒

This module is the bridge between a Parquet file’s thrift encoded metadata and this crate’s Parquet metadata API. It contains objects and functions used to serialize/deserialize metadata objects into/from the Thrift compact protocol format as defined by the Parquet specification.

writer 🔒

Structs

ColumnChunkMetaData

Metadata for a column chunk.

ColumnChunkMetaDataBuilder

Builder for ColumnChunkMetaData

ColumnIndexBuilder

Builder for Parquet ColumnIndex, part of the Parquet PageIndex

FileMetaData

File level metadata for a Parquet file.

FooterTail

Parsed Parquet footer tail (last 8 bytes of a Parquet file)

KeyValue

A key-value pair for FileMetaData.

LevelHistogram

Histograms for repetition and definition levels.

OffsetIndexBuilder

Builder for offset index, part of the Parquet PageIndex.

PageEncodingStats

PageEncodingStats for a column chunk and data page.

ParquetMetaData

Parsed metadata for a single Parquet file

ParquetMetaDataBuilder

A builder for creating / manipulating ParquetMetaData

ParquetMetaDataOptions

Options that can be set to control what parts of the Parquet file footer metadata will be decoded and made present in the ParquetMetaData returned by ParquetMetaDataReader and ParquetMetaDataPushDecoder.

ParquetMetaDataPushDecoder

A push decoder for ParquetMetaData.

ParquetMetaDataReader

Reads ParquetMetaData from a byte stream, with either synchronous or asynchronous I/O.

ParquetMetaDataWriter

Writes ParquetMetaData to a byte stream

RowGroupMetaData

Metadata for a row group

RowGroupMetaDataBuilder

Builder for row group metadata.

SortingColumn

Sort order within a RowGroup of a leaf column

Enums

PageIndexPolicy

Describes the policy for reading page indexes

ParquetPageEncodingStats 🔒

Internal representation of the page encoding stats in the ColumnChunkMetaData. This is not publicly exposed, with different getters defined for each variant.

ParquetStatisticsPolicy

Enum to control decoding of some Parquet statistics fields.

Type Aliases

FileMetaDataPtr

Reference counted pointer for FileMetaData.

RowGroupMetaDataPtr

Reference counted pointer for RowGroupMetaData.