Using pyarrow from C++ and Cython Code¶
pyarrow provides both a Cython and C++ API, allowing your own native code to interact with pyarrow objects.
C++ API¶
The Arrow C++ header files are bundled with a pyarrow installation.
To get the absolute path to this directory (like numpy.get_include()
), use:
import pyarrow as pa
pa.get_include()
Assuming the path above is on your compiler’s include path, the pyarrow API can be included using the following directive:
#include <arrow/python/pyarrow.h>
This will not include other parts of the Arrow API, which you will need
to include yourself (for example arrow/api.h
).
When building C extensions that use the Arrow C++ libraries, you must add
appropriate linker flags. We have provided functions pyarrow.get_libraries
and pyarrow.get_library_dirs
which return a list of library names and
likely library install locations (if you installed pyarrow with pip or
conda). These must be included when declaring your C extensions with
setuptools (see below).
Initializing the API¶
-
int import_pyarrow()¶
Initialize inner pointers of the pyarrow API. On success, 0 is returned. Otherwise, -1 is returned and a Python exception is set.
It is mandatory to call this function before calling any other function in the pyarrow C++ API. Failing to do so will likely lead to crashes.
Wrapping and Unwrapping¶
pyarrow provides the following functions to go back and forth between Python wrappers (as exposed by the pyarrow Python API) and the underlying C++ objects.
-
bool arrow::py::is_array(PyObject *obj)¶
Return whether obj wraps an Arrow C++
Array
pointer; in other words, whether obj is apyarrow.Array
instance.
-
bool arrow::py::is_batch(PyObject *obj)¶
Return whether obj wraps an Arrow C++
RecordBatch
pointer; in other words, whether obj is apyarrow.RecordBatch
instance.
-
bool arrow::py::is_buffer(PyObject *obj)¶
Return whether obj wraps an Arrow C++
Buffer
pointer; in other words, whether obj is apyarrow.Buffer
instance.
-
bool arrow::py::is_data_type(PyObject *obj)¶
Return whether obj wraps an Arrow C++
DataType
pointer; in other words, whether obj is apyarrow.DataType
instance.
-
bool arrow::py::is_field(PyObject *obj)¶
Return whether obj wraps an Arrow C++
Field
pointer; in other words, whether obj is apyarrow.Field
instance.
-
bool arrow::py::is_scalar(PyObject *obj)¶
Return whether obj wraps an Arrow C++
Scalar
pointer; in other words, whether obj is apyarrow.Scalar
instance.
-
bool arrow::py::is_schema(PyObject *obj)¶
Return whether obj wraps an Arrow C++
Schema
pointer; in other words, whether obj is apyarrow.Schema
instance.
-
bool arrow::py::is_table(PyObject *obj)¶
Return whether obj wraps an Arrow C++
Table
pointer; in other words, whether obj is apyarrow.Table
instance.
-
bool arrow::py::is_tensor(PyObject *obj)¶
Return whether obj wraps an Arrow C++
Tensor
pointer; in other words, whether obj is apyarrow.Tensor
instance.
-
bool arrow::py::is_sparse_coo_tensor(PyObject *obj)¶
Return whether obj wraps an Arrow C++
SparseCOOTensor
pointer; in other words, whether obj is apyarrow.SparseCOOTensor
instance.
-
bool arrow::py::is_sparse_csc_matrix(PyObject *obj)¶
Return whether obj wraps an Arrow C++
SparseCSCMatrix
pointer; in other words, whether obj is apyarrow.SparseCSCMatrix
instance.
-
bool arrow::py::is_sparse_csf_tensor(PyObject *obj)¶
Return whether obj wraps an Arrow C++
SparseCSFTensor
pointer; in other words, whether obj is apyarrow.SparseCSFTensor
instance.
-
bool arrow::py::is_sparse_csr_matrix(PyObject *obj)¶
Return whether obj wraps an Arrow C++
SparseCSRMatrix
pointer; in other words, whether obj is apyarrow.SparseCSRMatrix
instance.
The following functions expect a pyarrow object, unwrap the underlying
Arrow C++ API pointer, and return it as a Result
object. An error
may be returned if the input object doesn’t have the expected type.
-
Result<std::shared_ptr<Array>> arrow::py::unwrap_array(PyObject *obj)¶
Unwrap and return the Arrow C++
Array
pointer from obj.
-
Result<std::shared_ptr<RecordBatch>> arrow::py::unwrap_batch(PyObject *obj)¶
Unwrap and return the Arrow C++
RecordBatch
pointer from obj.
-
Result<std::shared_ptr<Buffer>> arrow::py::unwrap_buffer(PyObject *obj)¶
Unwrap and return the Arrow C++
Buffer
pointer from obj.
-
Result<std::shared_ptr<DataType>> arrow::py::unwrap_data_type(PyObject *obj)¶
Unwrap and return the Arrow C++
DataType
pointer from obj.
-
Result<std::shared_ptr<Field>> arrow::py::unwrap_field(PyObject *obj)¶
Unwrap and return the Arrow C++
Field
pointer from obj.
-
Result<std::shared_ptr<Scalar>> arrow::py::unwrap_scalar(PyObject *obj)¶
Unwrap and return the Arrow C++
Scalar
pointer from obj.
-
Result<std::shared_ptr<Schema>> arrow::py::unwrap_schema(PyObject *obj)¶
Unwrap and return the Arrow C++
Schema
pointer from obj.
-
Result<std::shared_ptr<Table>> arrow::py::unwrap_table(PyObject *obj)¶
Unwrap and return the Arrow C++
Table
pointer from obj.
-
Result<std::shared_ptr<Tensor>> arrow::py::unwrap_tensor(PyObject *obj)¶
Unwrap and return the Arrow C++
Tensor
pointer from obj.
-
Result<std::shared_ptr<SparseCOOTensor>> arrow::py::unwrap_sparse_coo_tensor(PyObject *obj)¶
Unwrap and return the Arrow C++
SparseCOOTensor
pointer from obj.
-
Result<std::shared_ptr<SparseCSCMatrix>> arrow::py::unwrap_sparse_csc_matrix(PyObject *obj)¶
Unwrap and return the Arrow C++
SparseCSCMatrix
pointer from obj.
-
Result<std::shared_ptr<SparseCSFTensor>> arrow::py::unwrap_sparse_csf_tensor(PyObject *obj)¶
Unwrap and return the Arrow C++
SparseCSFTensor
pointer from obj.
-
Result<std::shared_ptr<SparseCSRMatrix>> arrow::py::unwrap_sparse_csr_matrix(PyObject *obj)¶
Unwrap and return the Arrow C++
SparseCSRMatrix
pointer from obj.
The following functions take an Arrow C++ API pointer and wrap it in a pyarray object of the corresponding type. A new reference is returned. On error, NULL is returned and a Python exception is set.
Wrap the Arrow C++ array in a
pyarrow.Array
instance.
Wrap the Arrow C++ record batch in a
pyarrow.RecordBatch
instance.
Wrap the Arrow C++ buffer in a
pyarrow.Buffer
instance.
Wrap the Arrow C++ data_type in a
pyarrow.DataType
instance.
Wrap the Arrow C++ field in a
pyarrow.Field
instance.
Wrap the Arrow C++ scalar in a
pyarrow.Scalar
instance.
Wrap the Arrow C++ schema in a
pyarrow.Schema
instance.
Wrap the Arrow C++ table in a
pyarrow.Table
instance.
Wrap the Arrow C++ tensor in a
pyarrow.Tensor
instance.
Wrap the Arrow C++ sparse_tensor in a
pyarrow.SparseCOOTensor
instance.
Wrap the Arrow C++ sparse_tensor in a
pyarrow.SparseCSCMatrix
instance.
Wrap the Arrow C++ sparse_tensor in a
pyarrow.SparseCSFTensor
instance.
Wrap the Arrow C++ sparse_tensor in a
pyarrow.SparseCSRMatrix
instance.
Cython API¶
The Cython API more or less mirrors the C++ API, but the calling convention
can be different as required by Cython. In Cython, you don’t need to
initialize the API as that will be handled automatically by the cimport
directive.
Note
Classes from the Arrow C++ API are renamed when exposed in Cython, to
avoid named clashes with the corresponding Python classes. For example,
C++ Arrow arrays have the CArray
type and Array
is the
corresponding Python wrapper class.
Wrapping and Unwrapping¶
The following functions expect a pyarrow object, unwrap the underlying Arrow C++ API pointer, and return it. NULL is returned (without setting an exception) if the input is not of the right type.
- pyarrow.pyarrow_unwrap_batch(obj) shared_ptr[CRecordBatch] ¶
Unwrap the Arrow C++
RecordBatch
pointer from obj.
- pyarrow.pyarrow_unwrap_buffer(obj) shared_ptr[CBuffer] ¶
Unwrap the Arrow C++
Buffer
pointer from obj.
- pyarrow.pyarrow_unwrap_data_type(obj) shared_ptr[CDataType] ¶
Unwrap the Arrow C++
CDataType
pointer from obj.
- pyarrow.pyarrow_unwrap_scalar(obj) shared_ptr[CScalar] ¶
Unwrap the Arrow C++
Scalar
pointer from obj.
- pyarrow.pyarrow_unwrap_schema(obj) shared_ptr[CSchema] ¶
Unwrap the Arrow C++
Schema
pointer from obj.
- pyarrow.pyarrow_unwrap_tensor(obj) shared_ptr[CTensor] ¶
Unwrap the Arrow C++
Tensor
pointer from obj.
- pyarrow.pyarrow_unwrap_sparse_coo_tensor(obj) shared_ptr[CSparseCOOTensor] ¶
Unwrap the Arrow C++
SparseCOOTensor
pointer from obj.
- pyarrow.pyarrow_unwrap_sparse_csc_matrix(obj) shared_ptr[CSparseCSCMatrix] ¶
Unwrap the Arrow C++
SparseCSCMatrix
pointer from obj.
- pyarrow.pyarrow_unwrap_sparse_csf_tensor(obj) shared_ptr[CSparseCSFTensor] ¶
Unwrap the Arrow C++
SparseCSFTensor
pointer from obj.
- pyarrow.pyarrow_unwrap_sparse_csr_matrix(obj) shared_ptr[CSparseCSRMatrix] ¶
Unwrap the Arrow C++
SparseCSRMatrix
pointer from obj.
The following functions take a Arrow C++ API pointer and wrap it in a pyarray object of the corresponding type. An exception is raised on error.
- pyarrow.pyarrow_wrap_array(const shared_ptr[CArray]& array) object ¶
Wrap the Arrow C++ array in a Python
pyarrow.Array
instance.
- pyarrow.pyarrow_wrap_batch(const shared_ptr[CRecordBatch]& batch) object ¶
Wrap the Arrow C++ record batch in a Python
pyarrow.RecordBatch
instance.
- pyarrow.pyarrow_wrap_buffer(const shared_ptr[CBuffer]& buffer) object ¶
Wrap the Arrow C++ buffer in a Python
pyarrow.Buffer
instance.
- pyarrow.pyarrow_wrap_data_type(const shared_ptr[CDataType]& data_type) object ¶
Wrap the Arrow C++ data_type in a Python
pyarrow.DataType
instance.
- pyarrow.pyarrow_wrap_field(const shared_ptr[CField]& field) object ¶
Wrap the Arrow C++ field in a Python
pyarrow.Field
instance.
- pyarrow.pyarrow_wrap_resizable_buffer(const shared_ptr[CResizableBuffer]& buffer) object ¶
Wrap the Arrow C++ resizable buffer in a Python
pyarrow.ResizableBuffer
instance.
- pyarrow.pyarrow_wrap_scalar(const shared_ptr[CScalar]& scalar) object ¶
Wrap the Arrow C++ scalar in a Python
pyarrow.Scalar
instance.
- pyarrow.pyarrow_wrap_schema(const shared_ptr[CSchema]& schema) object ¶
Wrap the Arrow C++ schema in a Python
pyarrow.Schema
instance.
- pyarrow.pyarrow_wrap_table(const shared_ptr[CTable]& table) object ¶
Wrap the Arrow C++ table in a Python
pyarrow.Table
instance.
- pyarrow.pyarrow_wrap_tensor(const shared_ptr[CTensor]& tensor) object ¶
Wrap the Arrow C++ tensor in a Python
pyarrow.Tensor
instance.
- pyarrow.pyarrow_wrap_sparse_coo_tensor(const shared_ptr[CSparseCOOTensor]& sparse_tensor) object ¶
Wrap the Arrow C++ COO sparse tensor in a Python
pyarrow.SparseCOOTensor
instance.
- pyarrow.pyarrow_wrap_sparse_csc_matrix(const shared_ptr[CSparseCSCMatrix]& sparse_tensor) object ¶
Wrap the Arrow C++ CSC sparse tensor in a Python
pyarrow.SparseCSCMatrix
instance.
Example¶
The following Cython module shows how to unwrap a Python object and call the underlying C++ object’s API.
# distutils: language=c++
from pyarrow.lib cimport *
def get_array_length(obj):
# Just an example function accessing both the pyarrow Cython API
# and the Arrow C++ API
cdef shared_ptr[CArray] arr = pyarrow_unwrap_array(obj)
if arr.get() == NULL:
raise TypeError("not an array")
return arr.get().length()
To build this module, you will need a slightly customized setup.py
file
(this is assuming the file above is named example.pyx
):
from setuptools import setup
from Cython.Build import cythonize
import os
import numpy as np
import pyarrow as pa
ext_modules = cythonize("example.pyx")
for ext in ext_modules:
# The Numpy C headers are currently required
ext.include_dirs.append(np.get_include())
ext.include_dirs.append(pa.get_include())
ext.libraries.extend(pa.get_libraries())
ext.library_dirs.extend(pa.get_library_dirs())
if os.name == 'posix':
ext.extra_compile_args.append('-std=c++11')
# Try uncommenting the following line on Linux
# if you get weird linker errors or runtime crashes
# ext.define_macros.append(("_GLIBCXX_USE_CXX11_ABI", "0"))
setup(ext_modules=ext_modules)
Compile the extension:
python setup.py build_ext --inplace
Building Extensions against PyPI Wheels¶
The Python wheels have the Arrow C++ libraries bundled in the top level
pyarrow/
install directory. On Linux and macOS, these libraries have an ABI
tag like libarrow.so.17
which means that linking with -larrow
using the
linker path provided by pyarrow.get_library_dirs()
will not work right out
of the box. To fix this, you must run pyarrow.create_library_symlinks()
once as a user with write access to the directory where pyarrow is
installed. This function will attempt to create symlinks like
pyarrow/libarrow.so
. For example:
pip install pyarrow
python -c "import pyarrow; pyarrow.create_library_symlinks()"
Toolchain Compatibility (Linux)¶
The Python wheels for Linux are built using the PyPA manylinux images which use the CentOS devtoolset-9. In addition to the other notes above, if you are compiling C++ using these shared libraries, you will need to make sure you use a compatible toolchain as well or you might see a segfault during runtime.
Also, if you encounter errors when linking or loading the library, consider
setting the _GLIBCXX_USE_CXX11_ABI
preprocessor macro to 0
(for example by adding -D_GLIBCXX_USE_CXX11_ABI=0
to CFLAGS
).