Module arrow_writer 

Contains writer which writes arrow data into parquet data.

Re-exports

use crate::column::chunker::ContentDefinedChunker; 🔒

use bytes::Bytes; 🔒

use std::io::Write; 🔒

use std::ops::Range; 🔒

use std::slice::Iter; 🔒

use std::sync::Arc; 🔒

use std::sync::Mutex; 🔒

use std::vec::IntoIter; 🔒

use arrow_array::cast::AsArray; 🔒

use arrow_array::ArrayRef; 🔒

use arrow_array::Int32Array; 🔒

use arrow_array::RecordBatch; 🔒

use arrow_array::RecordBatchWriter; 🔒

use arrow_array::new_empty_array; 🔒

use arrow_array::PrimitiveArray; 🔒

use arrow_schema::ArrowError; 🔒

use arrow_schema::DataType as ArrowDataType; 🔒

use arrow_schema::Field; 🔒

use arrow_schema::FieldRef; 🔒

use arrow_schema::IntervalUnit; 🔒

use arrow_schema::SchemaRef; 🔒

use arrow_schema::TimeUnit; 🔒

use super::schema::add_encoded_arrow_schema_to_metadata; 🔒

use super::schema::decimal_length_from_precision; 🔒

use super::schema::validate_map_key_type; 🔒

use crate::arrow::ArrowSchemaConverter; 🔒

use crate::arrow::arrow_writer::byte_array::ByteArrayEncoder; 🔒

use crate::basic::PageType; 🔒

use crate::column::page::CompressedPage; 🔒

use crate::column::page::PageWriteSpec; 🔒

use crate::column::page::PageWriter; 🔒

use crate::column::page_encryption::PageEncryptor; 🔒

use crate::column::writer::encoder::ColumnValueEncoder; 🔒

use crate::column::writer::ColumnCloseResult; 🔒

use crate::column::writer::ColumnWriter; 🔒

use crate::column::writer::ColumnWriterImpl; 🔒

use crate::column::writer::GenericColumnWriter; 🔒

use crate::column::writer::get_column_writer; 🔒

use crate::data_type::ByteArray; 🔒

use crate::data_type::FixedLenByteArray; 🔒

use crate::data_type::FixedLenByteArrayType; 🔒

use std::collections::HashSet; 🔒

use crate::encryption::encrypt::FileEncryptor; 🔒

use crate::errors::ParquetError; 🔒

use crate::errors::Result; 🔒

use crate::file::metadata::KeyValue; 🔒

use crate::file::metadata::ParquetMetaData; 🔒

use crate::file::metadata::RowGroupMetaData; 🔒

use crate::file::properties::WriterProperties; 🔒

use crate::file::properties::WriterPropertiesPtr; 🔒

use crate::file::writer::SerializedFileWriter; 🔒

use crate::file::writer::SerializedRowGroupWriter; 🔒

use crate::parquet_thrift::ThriftCompactOutputProtocol; 🔒

use crate::parquet_thrift::WriteThrift; 🔒

use crate::schema::types::ColumnDescPtr; 🔒

use crate::schema::types::SchemaDescriptor; 🔒

use levels::ArrayLevels; 🔒

use levels::calculate_array_levels; 🔒

use arrow_array::types::*; 🔒

Modules

byte_array 🔒

levels 🔒

Parquet definition and repetition levels

Structs

ArrowColumnChunk

The data for a single column chunk, see ArrowColumnWriter

ArrowColumnChunkData 🔒

A single column chunk produced by ArrowColumnWriter.

ArrowColumnWriter

Encodes ArrowLeafColumn to ArrowColumnChunk

ArrowColumnWriterFactory 🔒

Creates ArrowColumnWriter instances

ArrowLeafColumn

A leaf column that can be encoded by ArrowColumnWriter

ArrowPageWriter 🔒

ArrowRowGroupWriter 🔒

Encodes [RecordBatch] to a parquet row group

ArrowRowGroupWriterFactory

Factory that creates new column writers for each row group in the Parquet file.

ArrowTopLevelWriterSpec 🔒

Associates a top-level Arrow field with its range of Parquet leaf writers.

ArrowWriteSchemaPlan 🔒

Immutable schema glue shared by every row group.

ArrowWriter

Encodes [RecordBatch] to parquet

ArrowWriterOptions

Arrow-specific configuration settings for writing parquet files.

InMemoryPageStore

The default PageStore, holding blobs on the heap in a Vec<Bytes>.

InMemoryPageStoreFactory

Factory for InMemoryPageStore — the default used by ArrowWriter.

PageKey

An opaque, store-allocated handle to a blob held by a PageStore.

PageStoreArgs

Context for a single PageStoreFactory::create call.

StreamingColumnChunkPages 🔒

A streaming iterator over one column chunk’s buffered page blobs, in final file order: the dictionary page (if any) first, then the data pages.

WriteFixedLenByteArray 🔒

Writes the values passed to it to a FIXED_LEN_BYTE_ARRAY column

WritePrimitive 🔒

Writes the values passed to it to an INT32 or INT64 column

Enums

ArrowColumnWriterImpl 🔒

ArrowWriteSchemaPlanError 🔒

Replayable schema-planning errors kept only for the deprecated ArrowRowGroupWriterFactory::new constructor. It cannot return errors, and repeated create_column_writers(&self) calls need fresh owned errors. ArrowRowGroupWriterFactory::try_new returns the original error directly.

Traits

FixedLenByteArrayVisitor 🔒

Receives the values ArrowWriter stores for an Arrow array in a FIXED_LEN_BYTE_ARRAY column, which are also the values hashed into the column’s bloom filter

NativeDecimalType 🔒

PageStore

A pluggable store for completed, serialized page blobs.

PageStoreFactory

Creates a fresh PageStore for each column chunk.

PrimitiveVisitor 🔒

Receives the values ArrowWriter stores for an Arrow array in an INT32 (T = i32) or INT64 (T = i64) column, which are also the values hashed into the column’s bloom filter

Functions

chunk_array_slice 🔒

chunk_contiguous_vec 🔒

compute_leaves

Computes the ArrowLeafColumn for a potentially nested [ArrayRef]

fixed_byte_width 🔒

Returns the fixed byte width for primitive Arrow types, or None for variable-length types.

get_bool_array_slice 🔒

hash_bytes 🔒

Hash a byte slice to a u64 for NDV tracking.

update_distinct_values_seen 🔒

Hash the non-null values in array (at non_null_indices) into seen.

visit_decimal 🔒

Passes the big-endian two’s complement bytes of the values of array to visitor

visit_fixed_len_byte_array 🔒

Passes the values ArrowWriter stores for column in a FIXED_LEN_BYTE_ARRAY column to visitor

visit_i32 🔒

Passes the values ArrowWriter stores for column in an INT32 column to visitor

visit_i64 🔒

Passes the values ArrowWriter stores for column in an INT64 column to visitor

write_leaf 🔒

write_primitive 🔒

Type Aliases

DistinctValuesSet 🔒

SharedColumnChunk 🔒

A shared ArrowColumnChunkData