Struct ParquetMetaData

pub struct ParquetMetaData {
    file_metadata: FileMetaData,
    row_groups: Vec<RowGroupMetaData>,
    column_index: Option<ParquetColumnIndex>,
    offset_index: Option<ParquetOffsetIndex>,
}

Parsed metadata for a single Parquet file

This structure is stored in the footer of Parquet files, in the format defined by parquet.thrift.

Overview

The fields of this structure are:

• FileMetaData: Information about the overall file (such as the schema) (See Self::file_metadata)
• RowGroupMetaData: Information about each Row Group (see Self::row_groups)
• ParquetColumnIndex and ParquetOffsetIndex: Optional “Page Index” structures (see Self::column_index and Self::offset_index)

This structure is read by the various readers in this crate or can be read directly from a file using the ParquetMetaDataReader struct.

See the ParquetMetaDataBuilder to create and modify this structure.

Fields

file_metadata: FileMetaData

File level metadata

row_groups: Vec<RowGroupMetaData>

Row group metadata

column_index: Option<ParquetColumnIndex>

Page level index for each page in each column chunk

offset_index: Option<ParquetOffsetIndex>

Offset index for each page in each column chunk

Implementations

impl ParquetMetaData

pub fn new( file_metadata: FileMetaData, row_groups: Vec<RowGroupMetaData>, ) -> Self

Creates Parquet metadata from file metadata and a list of row group metadata

pub fn new_with_page_index( file_metadata: FileMetaData, row_groups: Vec<RowGroupMetaData>, column_index: Option<ParquetColumnIndex>, offset_index: Option<ParquetOffsetIndex>, ) -> Self

👎Deprecated since 53.1.0: Use ParquetMetaDataBuilder

Creates Parquet metadata from file metadata, a list of row group metadata, and the column index structures.

pub fn into_builder(self) -> ParquetMetaDataBuilder

Convert this ParquetMetaData into a ParquetMetaDataBuilder

pub fn file_metadata(&self) -> &FileMetaData

Returns file metadata as reference.

pub fn num_row_groups(&self) -> usize

Returns number of row groups in this file.

pub fn row_group(&self, i: usize) -> &RowGroupMetaData

Returns row group metadata for ith position. Position should be less than number of row groups num_row_groups.

pub fn row_groups(&self) -> &[RowGroupMetaData]

Returns slice of row groups in this file.

pub fn column_index(&self) -> Option<&ParquetColumnIndex>

Returns the column index for this file if loaded

Returns None if the parquet file does not have a ColumnIndex or ArrowReaderOptions::with_page_index was set to false.

pub fn offset_index(&self) -> Option<&ParquetOffsetIndex>

Returns offset indexes in this file, if loaded

Returns None if the parquet file does not have a OffsetIndex or ArrowReaderOptions::with_page_index was set to false.

pub fn memory_size(&self) -> usize

Estimate of the bytes allocated to store ParquetMetadata

Notes:

1. Includes size of self

2. Includes heap memory for sub fields such as FileMetaData and RowGroupMetaData.

3. Includes memory from shared pointers (e.g. SchemaDescPtr). This means memory_size will over estimate the memory size if such pointers are shared.

4. Does not include any allocator overheads

pub(crate) fn set_column_index(&mut self, index: Option<ParquetColumnIndex>)

Override the column index

pub(crate) fn set_offset_index(&mut self, index: Option<ParquetOffsetIndex>)

Override the offset index

Trait Implementations

impl Clone for ParquetMetaData

fn clone(&self) -> ParquetMetaData

Returns a copy of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for ParquetMetaData

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl From<ParquetMetaData> for ParquetMetaDataBuilder

fn from(meta_data: ParquetMetaData) -> Self

Converts to this type from the input type.

impl PartialEq for ParquetMetaData

fn eq(&self, other: &ParquetMetaData) -> bool

Tests for self and other values to be equal, and is used by ==.

fn ne(&self, other: &Rhs) -> bool

Tests for !=. The default implementation is almost always sufficient, and should not be overridden without very good reason.

impl StructuralPartialEq for ParquetMetaData