PyArrow - Apache Arrow Python bindings¶
This is the documentation of the Python API of Apache Arrow.
Apache Arrow is a development platform for in-memory analytics. It contains a set of technologies that enable big data systems to store, process and move data fast.
See the parent documentation for additional details on the Arrow Project itself, on the Arrow format and the other language bindings.
The Arrow Python bindings (also named “PyArrow”) have first-class integration with NumPy, pandas, and built-in Python objects. They are based on the C++ implementation of Arrow.
Here we will detail the usage of the Python API for Arrow and the leaf libraries that add additional functionality such as reading Apache Parquet files into Arrow structures.
- Installing PyArrow
- Getting Started
- Data Types and In-Memory Data Model
- Compute Functions
- Memory and IO Interfaces
- Streaming, Serialization, and IPC
- Filesystem Interface
- Filesystem Interface (legacy)
- NumPy Integration
- Pandas Integration
- Dataframe Interchange Protocol
- Timestamps
- Reading and Writing the Apache ORC Format
- Reading and Writing CSV files
- Feather File Format
- Reading JSON files
- Reading and Writing the Apache Parquet Format
- Obtaining pyarrow with Parquet Support
- Reading and Writing Single Files
- Finer-grained Reading and Writing
- Inspecting the Parquet File Metadata
- Data Type Handling
- Compression, Encoding, and File Compatibility
- Partitioned Datasets (Multiple Files)
- Writing to Partitioned Datasets
- Reading from Partitioned Datasets
- Using with Spark
- Multithreaded Reads
- Reading from cloud storage
- Parquet Modular Encryption (Columnar Encryption)
- Tabular Datasets
- Arrow Flight RPC
- Extending pyarrow
- PyArrow Integrations
- Environment Variables
- API Reference
- Getting Involved
- Benchmarks