Pandas Integration

To interface with pandas, PyArrow provides various conversion routines to consume pandas structures and convert back to them.


While pandas uses NumPy as a backend, it has enough peculiarities (such as a different type system, and support for null values) that this is a separate topic from NumPy Integration.

To follow examples in this document, make sure to run:

In [1]: import pandas as pd

In [2]: import pyarrow as pa


The equivalent to a pandas DataFrame in Arrow is a Table. Both consist of a set of named columns of equal length. While pandas only supports flat columns, the Table also provides nested columns, thus it can represent more data than a DataFrame, so a full conversion is not always possible.

Conversion from a Table to a DataFrame is done by calling pyarrow.Table.to_pandas(). The inverse is then achieved by using pyarrow.Table.from_pandas().

import pyarrow as pa
import pandas as pd

df = pd.DataFrame({"a": [1, 2, 3]})
# Convert from pandas to Arrow
table = pa.Table.from_pandas(df)
# Convert back to pandas
df_new = table.to_pandas()

# Infer Arrow schema from pandas
schema = pa.Schema.from_pandas(df)


In Arrow, the most similar structure to a pandas Series is an Array. It is a vector that contains data of the same type as linear memory. You can convert a pandas Series to an Arrow Array using pyarrow.Array.from_pandas(). As Arrow Arrays are always nullable, you can supply an optional mask using the mask parameter to mark all null-entries.

Type differences

With the current design of pandas and Arrow, it is not possible to convert all column types unmodified. One of the main issues here is that pandas has no support for nullable columns of arbitrary type. Also datetime64 is currently fixed to nanosecond resolution. On the other side, Arrow might be still missing support for some types.

pandas -> Arrow Conversion

Source Type (pandas) Destination Type (Arrow)
bool BOOL
(u)int{8,16,32,64} (U)INT{8,16,32,64}
float32 FLOAT
float64 DOUBLE
str / unicode STRING
pd.Categorical DICTIONARY
pd.Timestamp TIMESTAMP(unit=ns) DATE

Arrow -> pandas Conversion

Source Type (Arrow) Destination Type (pandas)
BOOL bool
BOOL with nulls object (with values True, False, None)
(U)INT{8,16,32,64} (u)int{8,16,32,64}
(U)INT{8,16,32,64} with nulls float64
FLOAT float32
DOUBLE float64
DICTIONARY pd.Categorical
TIMESTAMP(unit=*) pd.Timestamp (np.datetime64[ns])
DATE object``(with `` objects)

Categorical types


Datetime (Timestamp) types


Date types

While dates can be handled using the datetime64[ns] type in pandas, some systems work with object arrays of Python’s built-in object:

In [3]: from datetime import date

In [4]: s = pd.Series([date(2018, 12, 31), None, date(2000, 1, 1)])

In [5]: s
0    2018-12-31
1          None
2    2000-01-01
dtype: object

When converting to an Arrow array, the date32 type will be used by default:

In [6]: arr = pa.array(s)

In [7]: arr.type
Out[7]: DataType(date32[day])

In [8]: arr[0]
Out[8]:, 12, 31)

To use the 64-bit date64, specify this explicitly:

In [9]: arr = pa.array(s, type='date64')

In [10]: arr.type
Out[10]: DataType(date64[ms])

When converting back with to_pandas, object arrays of objects are returned:

In [11]: arr.to_pandas()
array([, 12, 31), None,, 1, 1)],

If you want to use NumPy’s datetime64 dtype instead, pass date_as_object=False:

In [12]: s2 = pd.Series(arr.to_pandas(date_as_object=False))

In [13]: s2.dtype
Out[13]: dtype('<M8[ns]')

Time types