parquet/lib.rs
1// SPDX-License-Identifier: Apache-2.0
2// SPDX-FileCopyrightText: See the NOTICE file distributed with this work for additional information regarding copyright ownership
3// SPDX-FileContributor: Licensed to the Apache Software Foundation (ASF) under one or more contributor license agreements
4
5//!
6//! This crate contains the official Native Rust implementation of
7//! [Apache Parquet](https://parquet.apache.org/), part of
8//! the [Apache Arrow](https://arrow.apache.org/) project.
9//! The crate provides a number of APIs to read and write Parquet files,
10//! covering a range of use cases.
11//!
12//! Please see the [parquet crates.io](https://crates.io/crates/parquet)
13//! page for feature flags and tips to improve performance.
14//!
15//! # Format Overview
16//!
17//! Parquet is a columnar format, which means that unlike row formats like [CSV], values are
18//! iterated along columns instead of rows. Parquet is similar in spirit to [Arrow], but
19//! focuses on storage efficiency whereas Arrow prioritizes compute efficiency.
20//!
21//! Parquet files are partitioned for scalability. Each file contains metadata,
22//! along with zero or more "row groups", each row group containing one or
23//! more columns. The APIs in this crate reflect this structure.
24//!
25//! Data in Parquet files is strongly typed and differentiates between logical
26//! and physical types (see [`schema`]). In addition, Parquet files may contain
27//! other metadata, such as statistics, which can be used to optimize reading
28//! (see [`file::metadata`]).
29//! For more details about the Parquet format itself, see the [Parquet spec]
30//!
31//! [Parquet spec]: https://github.com/apache/parquet-format/blob/master/README.md#file-format
32//!
33//! # APIs
34//!
35//! This crate exposes a number of APIs for different use-cases.
36//!
37//! ## Metadata and Schema
38//!
39//! The [`schema`] module provides APIs to work with Parquet schemas. The
40//! [`file::metadata`] module provides APIs to work with Parquet metadata.
41//!
42//! ## Reading and Writing Arrow (`arrow` feature)
43//!
44//! The [`arrow`] module supports reading and writing Parquet data to/from
45//! Arrow [`RecordBatch`]es. Using Arrow is simple and performant, and allows workloads
46//! to leverage the wide range of data transforms provided by the [arrow] crate, and by the
47//! ecosystem of [Arrow] compatible systems.
48//!
49//! Most users will use [`ArrowWriter`] for writing and [`ParquetRecordBatchReaderBuilder`] for
50//! reading from synchronous IO sources such as files or in-memory buffers.
51//!
52//! Lower level APIs include
53//! * [`ParquetPushDecoder`] for file grained control over interleaving of IO and CPU.
54//! * [`ArrowColumnWriter`] for writing using multiple threads,
55//! * [`RowFilter`] to apply filters during decode
56//!
57//! ### EXPERIMENTAL: Content-Defined Chunking
58//!
59//! [`ArrowWriter`] supports content-defined chunking (CDC), which creates data page
60//! boundaries based on content rather than fixed sizes. CDC enables efficient
61//! deduplication in content-addressable storage (CAS) systems: when the same data
62//! appears in successive file versions, it will produce identical byte sequences that
63//! CAS backends can deduplicate.
64//!
65//! Enable CDC via [`WriterProperties`]:
66//!
67//! ```rust
68//! # use parquet::file::properties::{WriterProperties, CdcOptions};
69//! let props = WriterProperties::builder()
70//! .set_content_defined_chunking(Some(CdcOptions::default()))
71//! .build();
72//! ```
73//!
74//! See [`CdcOptions`] for chunk size and normalization parameters.
75//!
76//! [`WriterProperties`]: file::properties::WriterProperties
77//! [`CdcOptions`]: file::properties::CdcOptions
78//!
79//! [`ArrowWriter`]: arrow::arrow_writer::ArrowWriter
80//! [`ParquetRecordBatchReaderBuilder`]: arrow::arrow_reader::ParquetRecordBatchReaderBuilder
81//! [`ParquetPushDecoder`]: arrow::push_decoder::ParquetPushDecoder
82//! [`ArrowColumnWriter`]: arrow::arrow_writer::ArrowColumnWriter
83//! [`RowFilter`]: arrow::arrow_reader::RowFilter
84//!
85//! ## `async` Reading and Writing Arrow (`arrow` feature + `async` feature)
86//!
87//! The [`async_reader`] and [`async_writer`] modules provide async APIs to
88//! read and write [`RecordBatch`]es asynchronously.
89//!
90//! Most users will use [`AsyncArrowWriter`] for writing and [`ParquetRecordBatchStreamBuilder`]
91//! for reading, automatically optimizing IO based on any predicates or projections provided.
92//! Object storage services such as S3 can be integrated by implementing
93//! [`AsyncFileReader`] on top of a client such as the [object_store] crate,
94//! or by passing a writer implementing [`AsyncWrite`] (such as
95//! `object_store::buffered::BufWriter`) to [`AsyncArrowWriter`].
96//!
97//! [`async_reader`]: arrow::async_reader
98//! [`async_writer`]: arrow::async_writer
99//! [`AsyncArrowWriter`]: arrow::async_writer::AsyncArrowWriter
100//! [`AsyncFileReader`]: arrow::async_reader::AsyncFileReader
101//! [`AsyncWrite`]: https://docs.rs/tokio/latest/tokio/io/trait.AsyncWrite.html
102//! [`ParquetRecordBatchStreamBuilder`]: arrow::async_reader::ParquetRecordBatchStreamBuilder
103//!
104//! ## Variant Logical Type (`variant_experimental` feature)
105//!
106//! The [`variant`] module supports reading and writing Parquet files
107//! with the [Variant Binary Encoding] logical type, which can represent
108//! semi-structured data such as JSON efficiently.
109//!
110//! [Variant Binary Encoding]: https://github.com/apache/parquet-format/blob/master/VariantEncoding.md
111//!
112//! ## FILE Logical Type (`file_experimental` feature)
113//!
114//! The [`FileType`] Arrow extension type maps to the Parquet [FILE] logical
115//! type: a `FILE`-annotated group is read as a struct with this extension
116//! type, and a struct field with this extension type is written as a
117//! `FILE`-annotated group.
118//!
119//! [`FileType`]: arrow::file_type::FileType
120//! [FILE]: https://github.com/apache/parquet-format/blob/master/LogicalTypes.md#file
121//!
122//! ## Read/Write Parquet Directly
123//!
124//! Workloads needing finer-grained control, or to avoid a dependence on arrow,
125//! can use the APIs in [`mod@file`] directly. These APIs are harder to use
126//! as they directly use the underlying Parquet data model, and require knowledge
127//! of the Parquet format, including the details of [Dremel] record shredding
128//! and [Logical Types].
129//!
130//! [arrow]: https://docs.rs/arrow/latest/arrow/index.html
131//! [Arrow]: https://arrow.apache.org/
132//! [`RecordBatch`]: https://docs.rs/arrow/latest/arrow/array/struct.RecordBatch.html
133//! [CSV]: https://en.wikipedia.org/wiki/Comma-separated_values
134//! [Dremel]: https://research.google/pubs/pub36632/
135//! [Logical Types]: https://github.com/apache/parquet-format/blob/master/LogicalTypes.md
136//! [object_store]: https://docs.rs/object_store/latest/object_store/
137//!
138//! # Platform Support
139//!
140//! Only little-endian platforms are officially supported and tested in CI.
141//! Big-endian platforms are not tested in CI and may not work correctly.
142//! Fixes for big-endian platforms are welcome and handled on a best-effort basis,
143//! but compatibility is not guaranteed.
144
145#![doc(
146 html_logo_url = "https://raw.githubusercontent.com/apache/parquet-format/25f05e73d8cd7f5c83532ce51cb4f4de8ba5f2a2/logo/parquet-logos_1.svg",
147 html_favicon_url = "https://raw.githubusercontent.com/apache/parquet-format/25f05e73d8cd7f5c83532ce51cb4f4de8ba5f2a2/logo/parquet-logos_1.svg"
148)]
149#![cfg_attr(docsrs, feature(doc_cfg))]
150#![warn(missing_docs)]
151#[cfg(all(
152 feature = "flate2",
153 not(any(feature = "flate2-zlib-rs", feature = "flate2-rust_backend"))
154))]
155compile_error!(
156 "When enabling `flate2` you must enable one of the features: `flate2-zlib-rs` or `flate2-rust_backend`."
157);
158
159#[macro_use]
160pub mod errors;
161pub mod basic;
162
163#[macro_use]
164pub mod data_type;
165
166use std::fmt::Debug;
167use std::ops::Range;
168// Exported for external use, such as benchmarks
169#[cfg(feature = "experimental")]
170#[doc(hidden)]
171pub use self::encodings::{decoding, encoding};
172
173// Keep these module declarations explicit: rustfmt does not discover modules declared by macros.
174// See https://github.com/rust-lang/rustfmt/issues/3253
175#[cfg(feature = "experimental")]
176#[doc(hidden)]
177#[macro_use]
178pub mod util;
179#[cfg(not(feature = "experimental"))]
180#[macro_use]
181mod util;
182
183pub use util::utf8;
184
185#[cfg(feature = "arrow")]
186pub mod arrow;
187pub mod bloom_filter;
188pub mod column;
189#[cfg(feature = "experimental")]
190#[doc(hidden)]
191pub mod compression;
192#[cfg(not(feature = "experimental"))]
193mod compression;
194#[cfg(feature = "experimental")]
195#[doc(hidden)]
196pub mod encodings;
197#[cfg(not(feature = "experimental"))]
198mod encodings;
199
200#[cfg(feature = "encryption")]
201#[cfg_attr(feature = "experimental", doc(hidden))]
202pub mod encryption;
203
204pub mod file;
205pub mod record;
206pub mod schema;
207
208mod parquet_macros;
209mod parquet_thrift;
210
211/// What data is needed to read the next item from a decoder.
212///
213/// This is used to communicate between the decoder and the caller
214/// to indicate what data is needed next, or what the result of decoding is.
215#[derive(Debug)]
216pub enum DecodeResult<T: Debug> {
217 /// The ranges of data necessary to proceed
218 // TODO: distinguish between minimum needed to make progress and what could be used?
219 NeedsData(Vec<Range<u64>>),
220 /// The decoder produced an output item
221 Data(T),
222 /// The decoder finished processing
223 Finished,
224}
225
226#[cfg_attr(feature = "experimental", doc(hidden))]
227pub mod geospatial;
228#[cfg(feature = "variant_experimental")]
229pub mod variant;