Crate arrow_pyarrow 

Pass Arrow objects from and to PyArrow, using Arrow’s C Data Interface and pyo3.

For underlying implementation, see the [ffi] module.

One can use these to write Python functions that take and return PyArrow objects, with automatic conversion to corresponding arrow-rs types.

#[pyfunction]
fn double_array(array: PyArrowType<ArrayData>) -> PyResult<PyArrowType<ArrayData>> {
    let array = array.0; // Extract from PyArrowType wrapper
    let array: Arc<dyn Array> = make_array(array); // Convert ArrayData to ArrayRef
    let array: &Int32Array = array.as_any().downcast_ref()
        .ok_or_else(|| PyValueError::new_err("expected int32 array"))?;
    let array: Int32Array = array.iter().map(|x| x.map(|x| x * 2)).collect();
    Ok(PyArrowType(array.into_data()))
}

pyarrow type	arrow-rs type
pyarrow.DataType	[DataType]
pyarrow.Field	[Field]
pyarrow.Schema	[Schema]
pyarrow.Array	[ArrayData]
pyarrow.RecordBatch	[RecordBatch]
pyarrow.RecordBatchReader	[RecordBatchStreamReader] / Box<dyn RecordBatchReader + Send> (1)
pyarrow.Table	Table (2)

(1) pyarrow.RecordBatchReader can be imported as [RecordBatchStreamReader]. Either [RecordBatchStreamReader] or Box<dyn RecordBatchReader + Send> can be exported as pyarrow.RecordBatchReader. (Box<dyn RecordBatchReader + Send> is typically easier to create.)

(2) Although arrow-rs offers Table, a convenience wrapper for pyarrow.Table that internally holds Vec<RecordBatch>, it is meant primarily for use cases where you already have Vec<RecordBatch> on the Rust side and want to export that in bulk as a pyarrow.Table. In general, it is recommended to use streaming approaches instead of dealing with data in bulk. For example, a pyarrow.Table (or any other object that implements the ArrayStream PyCapsule interface) can be imported to Rust through PyArrowType<RecordBatchStreamReader> instead of forcing eager reading into Vec<RecordBatch>.

Type stubs

With the experimental-inspect feature enabled, each conversion records the pyarrow class it maps to in PyO3’s introspection data, so a generated .pyi says pyarrow.Array where it would otherwise say _typeshed.Incomplete. The hints live on FromPyArrow::INPUT_TYPE, ToPyArrow::OUTPUT_TYPE and IntoPyArrow::OUTPUT_TYPE, and PyArrowType forwards them.

Input hints name the pyarrow classes only, and are therefore narrower than what is accepted: the PyCapsule interface is duck-typed and has no canonical Python type to name.

Platform Support

Only little-endian platforms are officially supported and tested in CI. Big-endian platforms are not tested in CI and may not work correctly. Fixes for big-endian platforms are welcome and handled on a best-effort basis, but compatibility is not guaranteed.

Re-exports

pub(crate) use From; 🔒

pub(crate) use TryFrom; 🔒

pub(crate) use std::ffi::CStr; 🔒

pub(crate) use std::ptr::NonNull; 🔒

pub(crate) use std::sync::Arc; 🔒

pub(crate) use arrow_array::ffi; 🔒

pub(crate) use FFI_ArrowArray; 🔒

pub(crate) use FFI_ArrowSchema; 🔒

pub(crate) use FFI_ArrowArrayStream; 🔒

pub(crate) use RecordBatchStreamReader; 🔒

pub(crate) use RecordBatch; 🔒

pub(crate) use RecordBatchIterator; 🔒

pub(crate) use RecordBatchOptions; 🔒

pub(crate) use RecordBatchReader; 🔒

pub(crate) use StructArray; 🔒

pub(crate) use make_array; 🔒

pub(crate) use arrow_data::ArrayData; 🔒

pub(crate) use ArrowError; 🔒

pub(crate) use DataType; 🔒

pub(crate) use Field; 🔒

pub(crate) use Schema; 🔒

pub(crate) use SchemaRef; 🔒

pub(crate) use PyTypeError; 🔒

pub(crate) use PyValueError; 🔒

pub(crate) use pyo3::ffi::Py_uintptr_t; 🔒

pub(crate) use pyo3::sync::PyOnceLock; 🔒

pub(crate) use PyCapsule; 🔒

pub(crate) use PyDict; 🔒

pub(crate) use PyList; 🔒

pub(crate) use PyString; 🔒

pub(crate) use PyType; 🔒

pub(crate) use CastError; 🔒

pub(crate) use import_exception; 🔒

pub(crate) use intern; 🔒

pub(crate) use inspect::PyStaticExpr; 🔒

pub(crate) use type_hint_identifier; 🔒

pub(crate) use type_hint_subscript; 🔒

pub(crate) use type_hint_union; 🔒

pub(crate) use type_object::PyTypeInfo; 🔒

pub(crate) use pyo3::prelude::*; 🔒

Macros

type_hint 🔒

Declares a FromPyArrow::INPUT_TYPE / ToPyArrow::OUTPUT_TYPE / IntoPyArrow::OUTPUT_TYPE hint on an impl, expanding to nothing unless the experimental-inspect feature is enabled.

Structs

ArrowException

A Rust type representing an exception defined in Python code.

PyArrowType

A newtype wrapper for types implementing FromPyArrow or IntoPyArrow.

Table

This is a convenience wrapper around Vec<RecordBatch> that tries to simplify conversion from and to pyarrow.Table.

Constants

ARRAY_STREAM_INPUT_TYPE 🔒

The type hint shared by every conversion that imports through the ArrowArrayStream PyCapsule interface, i.e. [RecordBatchStreamReader] and Table.

Traits

FromPyArrow

Trait for converting Python objects to arrow-rs types.

FromPyArrowUnchecked

Trait for converting Python objects to arrow-rs types without validation.

IntoPyArrow

Convert an arrow-rs type into a PyArrow object.

ToPyArrow

Create a new PyArrow object from a arrow-rs type.

Functions

array_class 🔒

call_arrow_c_array_method_if_exists 🔒

call_capsule_method_if_exists 🔒

data_type_class 🔒

extract_capsule 🔒

field_class 🔒

record_batch_class 🔒

record_batch_from_pyarrow_bound_impl 🔒

record_batch_reader_class 🔒

schema_class 🔒

table_class 🔒

to_py_err 🔒

validate_class 🔒

wrapping_type_error 🔒

Type Aliases

PyArrowException

Represents an exception raised by PyArrow.