Skip to main content

Module metadata

parquet::file

Module metadata 

Source
Expand description

Parquet metadata API

Users should use these structures to interact with Parquet metadata.

  • ParquetMetaData: Top level metadata container, read from the Parquet file footer.

  • FileMetaData: File level metadata such as schema, row counts and version.

  • RowGroupMetaData: Metadata for each Row Group with a File, such as location and number of rows, and column chunks.

  • ColumnChunkMetaData: Metadata for each column chunk (primitive leaf) within a Row Group including encoding and compression information, number of values, statistics, etc.

  • PageIndex: Metadata and statistics used to reduce page-level I/O.

§APIs for working with Parquet Metadata

The Parquet readers and writers in this crate handle reading and writing metadata into Parquet files. To work with metadata directly, the following APIs are available:

§Examples

Please see external_metadata.rs

Re-exports§

use crate::basic::BoundaryOrder; 🔒
use crate::basic::ColumnOrder; 🔒
use crate::basic::Compression; 🔒
use crate::basic::CompressionCodec; 🔒
use crate::basic::Encoding; 🔒
use crate::basic::EncodingMask; 🔒
use crate::basic::PageType; 🔒
use crate::basic::Type; 🔒
use crate::encryption::decrypt::FileDecryptor; 🔒
use crate::errors::ParquetError; 🔒
use crate::errors::Result; 🔒
use crate::file::column_crypto_metadata::ColumnCryptoMetaData; 🔒
pub(crate) use crate::file::metadata::memory::HeapSize; 🔒
use crate::file::metadata::page_index::PageIndex; 🔒
use crate::file::metadata::page_index::PageIndexProvider; 🔒
use crate::file::metadata::page_index::RowGroupPageIndex; 🔒
use crate::file::metadata::thrift::encryption::EncryptionAlgorithm; 🔒
use crate::file::page_index::column_index::ByteArrayColumnIndex; 🔒
use crate::file::page_index::column_index::PrimitiveColumnIndex; 🔒
use crate::file::page_index::column_index::ColumnIndexMetaData; 🔒
use crate::file::page_index::offset_index::PageLocation; 🔒
use crate::file::statistics::Statistics; 🔒
use crate::geospatial::statistics as geo_statistics; 🔒
use crate::parquet_thrift::ElementType; 🔒
use crate::parquet_thrift::FieldType; 🔒
use crate::parquet_thrift::ReadThrift; 🔒
use crate::parquet_thrift::ThriftCompactInputProtocol; 🔒
use crate::parquet_thrift::ThriftCompactOutputProtocol; 🔒
use crate::parquet_thrift::WriteThrift; 🔒
use crate::parquet_thrift::WriteThriftField; 🔒
use crate::schema::types::ColumnDescPtr; 🔒
use crate::schema::types::ColumnDescriptor; 🔒
use crate::schema::types::ColumnPath; 🔒
use crate::schema::types::SchemaDescPtr; 🔒
use crate::schema::types::SchemaDescriptor; 🔒
use crate::schema::types::Type as SchemaType; 🔒
use crate::data_type::private::ParquetValueType; 🔒
use crate::file::page_index::offset_index::OffsetIndexMetaData; 🔒
pub use footer_tail::FooterTail;
pub use options::ParquetMetaDataOptions;
pub use options::ParquetStatisticsPolicy;
pub use push_decoder::ParquetMetaDataPushDecoder;
pub use reader::ColumnChunkMask;
pub use reader::PageIndexPolicy;
pub use reader::ParquetMetaDataReader;
use std::io::Write; 🔒
use std::ops::Range; 🔒
use std::sync::Arc; 🔒
pub use writer::ParquetMetaDataWriter;
pub(crate) use writer::ThriftMetadataWriter; 🔒

Modules§

dictionary 🔒
Decoding a column chunk’s dictionary page directly into an Arrow array, independent of the row-by-row ArrayReader machinery.
footer_tail 🔒
memory 🔒
Memory calculations for ParquetMetadata::memory_size
options 🔒
Options used to control metadata parsing
page_index
Page Index structures for efficient page-level skipping
parser 🔒
Internal metadata parsing routines
push_decoder 🔒
reader 🔒
thrift 🔒
This module is the bridge between a Parquet file’s thrift encoded metadata and this crate’s Parquet metadata API. It contains objects and functions used to serialize/deserialize metadata objects into/from the Thrift compact protocol format as defined by the Parquet specification.
writer 🔒

Structs§

ColumnChunkMetaData
Metadata for a column chunk.
ColumnChunkMetaDataBuilder
Builder for ColumnChunkMetaData
ColumnIndexBuilder
Builder for Parquet ColumnIndex, part of the Parquet PageIndex
FileMetaData
File level metadata for a Parquet file.
KeyValue
A key-value pair for FileMetaData.
LevelHistogram
Histograms for repetition and definition levels.
OffsetIndexBuilder
Builder for offset index, part of the Parquet PageIndex.
PageEncodingStats
PageEncodingStats for a column chunk and data page.
ParquetMetaData
Parsed metadata for a single Parquet file
ParquetMetaDataBuilder
A builder for creating / manipulating ParquetMetaData
RowGroupMetaData
Metadata for a row group
RowGroupMetaDataBuilder
Builder for row group metadata.
SortingColumn
Sort order within a RowGroup of a leaf column

Enums§

ParquetPageEncodingStats 🔒
Internal representation of the page encoding stats in the ColumnChunkMetaData. This is not publicly exposed, with different getters defined for each variant.
RowGroups 🔒

Type Aliases§

FileMetaDataPtr
Reference counted pointer for FileMetaData.
RowGroupMetaDataPtr
Reference counted pointer for RowGroupMetaData.