Struct ParquetMetaDataWriter 

pub struct ParquetMetaDataWriter<'a, W: Write> {
    buf: TrackedWrite<W>,
    metadata: &'a ParquetMetaData,
    write_path_in_schema: bool,
    preserve_page_index_locations: bool,
}

Writes ParquetMetaData to a byte stream

This structure handles the details of writing the various parts of Parquet metadata into a byte stream. It is used to write the metadata into a parquet file and can also write metadata into other locations (such as a store of bytes).

Discussion

The process of writing Parquet metadata is tricky because the metadata is not stored as a single inline thrift structure. It can have several “out of band” structures such as the OffsetIndex and BloomFilters stored in separate structures whose locations are stored as offsets from the beginning of the file.

Note: this writer does not directly write BloomFilters. In order to write BloomFilters, write the bloom filters into the buffer before creating the metadata writer. Then set the corresponding bloom_filter_offset and bloom_filter_length on ColumnChunkMetaData passed to this writer.

The writer serializes the page index of any PageIndexProvider attached to the input ParquetMetaData. By default, if the provider has no index for a column chunk, the offset and length of that index in the ColumnChunkMetaData are preserved in the output metadata. This is done so one can store the footer metadata externally but keep the page indexes in the original file. If this behavior is not desired, then Self::with_preserve_page_index_locations should be set to false.

Output Format

The format of the metadata is as follows:

1. Optional ColumnIndex (thrift encoded)
2. Optional OffsetIndex (thrift encoded)
3. FileMetaData (thrift encoded)
4. Length of encoded FileMetaData (4 bytes, little endian)
5. Parquet Magic Bytes (4 bytes)

┌──────────────────────┐
│                      │
│         ...          │
│                      │
│┌ ─ ─ ─ ─ ─ ─ ─ ─ ─ ┐ │
│     ColumnIndex     ◀│─ ─ ─
││    (Optional)     │ │     │
│ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─  │
│┌ ─ ─ ─ ─ ─ ─ ─ ─ ─ ┐ │     │ FileMetadata
│     OffsetIndex      │       contains embedded
││    (Optional)     │◀┼ ─   │ offsets to
│ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─  │  │    ColumnIndex and
│╔═══════════════════╗ │     │ OffsetIndex
│║                   ║ │  │
│║                   ║ ┼ ─   │
│║   FileMetadata    ║ │
│║                   ║ ┼ ─ ─ ┘
│║                   ║ │
│╚═══════════════════╝ │
│┌───────────────────┐ │
││  metadata length  │ │ length of FileMetadata  (only)
│└───────────────────┘ │
│┌───────────────────┐ │
││      'PAR1'       │ │ Parquet Magic Bytes
│└───────────────────┘ │
└──────────────────────┘
     Output Buffer

Example

// write parquet metadata to an in-memory buffer
let mut buffer = vec![];
let metadata: ParquetMetaData = get_metadata();
let writer = ParquetMetaDataWriter::new(&mut buffer, &metadata);
// write the metadata to the buffer
writer.finish().unwrap();
assert!(!buffer.is_empty());

Fields

buf: TrackedWrite<W>

metadata: &'a ParquetMetaData

write_path_in_schema: bool

preserve_page_index_locations: bool

Implementations

impl<'a, W: Write> ParquetMetaDataWriter<'a, W>

pub fn new(buf: W, metadata: &'a ParquetMetaData) -> Self

Create a new ParquetMetaDataWriter to write to buf

Note any embedded offsets in the metadata will be written assuming the metadata is at the start of the buffer. If the metadata is being written to a location other than the start of the buffer, see Self::new_with_tracked

See example on the struct level documentation

pub fn new_with_tracked( buf: TrackedWrite<W>, metadata: &'a ParquetMetaData, ) -> Self

Create a new ParquetMetaDataWriter to write to buf

This method is used when the metadata is being written to a location other than the start of the buffer.

See example on the struct level documentation

pub fn with_write_path_in_schema(self, val: bool) -> Self

Set whether or not to write the path_in_schema field in the Thrift ColumnMetaData struct.

pub fn with_preserve_page_index_locations(self, val: bool) -> Self

Set whether or not to preserve the page index location metadata in the Thrift ColumnMetaData (defaults to true).

Because this struct is often used to externalize the footer metadata, it is usually desirable to preserve this location information, even when the page indexes are not duplicated (for instance if the provided ParquetMetaData returns no PageIndexProvider). As such, this defaults to true.

Set this to false to reset the location metadata if no page indexes are present in the provided ParquetMetaData.

pub fn finish(self) -> Result<()>

Write the metadata to the buffer