Struct SerializedFileWriter

pub struct SerializedFileWriter<W: Write> {
    buf: TrackedWrite<W>,
    schema: TypePtr,
    descr: SchemaDescPtr,
    props: WriterPropertiesPtr,
    row_groups: Vec<RowGroupMetaData>,
    bloom_filters: Vec<Vec<Option<Sbbf>>>,
    column_indexes: Vec<Vec<Option<ColumnIndex>>>,
    offset_indexes: Vec<Vec<Option<OffsetIndex>>>,
    row_group_index: usize,
    kv_metadatas: Vec<KeyValue>,
    finished: bool,
    file_encryptor: Option<Arc<FileEncryptor>>,
}

Parquet file writer API. Provides methods to write row groups sequentially.

The main workflow should be as following:

• Create file writer, this will open a new file and potentially write some metadata.
• Request a new row group writer by calling next_row_group.
• Once finished writing row group, close row group writer by calling close
• Write subsequent row groups, if necessary.
• After all row groups have been written, close the file writer using close method.

Fields

buf: TrackedWrite<W>

schema: TypePtr

descr: SchemaDescPtr

props: WriterPropertiesPtr

row_groups: Vec<RowGroupMetaData>

bloom_filters: Vec<Vec<Option<Sbbf>>>

column_indexes: Vec<Vec<Option<ColumnIndex>>>

offset_indexes: Vec<Vec<Option<OffsetIndex>>>

row_group_index: usize

kv_metadatas: Vec<KeyValue>

finished: bool

file_encryptor: Option<Arc<FileEncryptor>>

Implementations

impl<W: Write + Send> SerializedFileWriter<W>

pub fn new( buf: W, schema: TypePtr, properties: WriterPropertiesPtr, ) -> Result<Self>

Creates new file writer.

fn get_file_encryptor( properties: &WriterPropertiesPtr, schema_descriptor: &SchemaDescriptor, ) -> Result<Option<Arc<FileEncryptor>>>

pub fn next_row_group(&mut self) -> Result<SerializedRowGroupWriter<'_, W>>

Creates new row group from this file writer. In case of IO error or Thrift error, returns Err.

There can be at most 2^15 row groups in a file; and row groups have to be written sequentially. Every time the next row group is requested, the previous row group must be finalised and closed using RowGroupWriter::close method.

pub fn flushed_row_groups(&self) -> &[RowGroupMetaData]

Returns metadata for any flushed row groups

pub fn finish(&mut self) -> Result<FileMetaData>

Close and finalize the underlying Parquet writer

Unlike Self::close this does not consume self

Attempting to write after calling finish will result in an error

pub fn close(self) -> Result<FileMetaData>

Closes and finalises file writer, returning the file metadata.

fn start_file( properties: &WriterPropertiesPtr, buf: &mut TrackedWrite<W>, ) -> Result<()>

Writes magic bytes at the beginning of the file.

fn write_metadata(&mut self) -> Result<FileMetaData>

Assembles and writes metadata at the end of the file.

fn assert_previous_writer_closed(&self) -> Result<()>

pub fn append_key_value_metadata(&mut self, kv_metadata: KeyValue)

Add a KeyValue to the file writer’s metadata

pub fn schema_descr(&self) -> &SchemaDescriptor

Returns a reference to schema descriptor.

pub fn properties(&self) -> &WriterPropertiesPtr

Returns a reference to the writer properties

pub fn inner(&self) -> &W

Returns a reference to the underlying writer.

pub fn inner_mut(&mut self) -> &mut W

Returns a mutable reference to the underlying writer.

It is inadvisable to directly write to the underlying writer.

pub fn into_inner(self) -> Result<W>

Writes the file footer and returns the underlying writer.

pub fn bytes_written(&self) -> usize

Returns the number of bytes written to this instance

pub(crate) fn file_encryptor(&self) -> Option<Arc<FileEncryptor>>

Get the file encryptor used by this instance to encrypt data

Trait Implementations

impl<W: Write> Debug for SerializedFileWriter<W>

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.