Dataframe Interchange Protocol

The interchange protocol is implemented for pa.Table and pa.RecordBatch and is used to interchange data between PyArrow and other dataframe libraries that also have the protocol implemented. The data structures that are supported in the protocol are primitive data types plus the dictionary data type. The protocol also has missing data support and it supports chunking, meaning accessing the data in “batches” of rows.

The Python dataframe interchange protocol is designed by the Consortium for Python Data API Standards in order to enable data interchange between dataframe libraries in the Python ecosystem. See more about the standard in the protocol documentation.

From pyarrow to other libraries: __dataframe__() method

The __dataframe__() method creates a new exchange object that the consumer library can take and construct an object of it’s own.

>>> import pyarrow as pa
>>> table = pa.table({"n_atendees": [100, 10, 1]})
>>> table.__dataframe__()
<pyarrow.interchange.dataframe._PyArrowDataFrame object at ...>

This is meant to be used by the consumer library when calling the from_dataframe() function and is not meant to be used manually by the user.

From other libraries to pyarrow: from_dataframe()

With the from_dataframe() function, we can construct a pyarrow.Table from any dataframe object that implements the __dataframe__() method via the dataframe interchange protocol.

We can for example take a pandas dataframe and construct a pyarrow table with the use of the interchange protocol:

>>> import pyarrow
>>> from pyarrow.interchange import from_dataframe

>>> import pandas as pd
>>> df = pd.DataFrame({
...         "n_atendees": [100, 10, 1],
...         "country": ["Italy", "Spain", "Slovenia"],
...     })
>>> df
   n_atendees   country
0         100     Italy
1          10     Spain
2           1  Slovenia
>>> from_dataframe(df)
pyarrow.Table
n_atendees: int64
country: large_string
----
n_atendees: [[100,10,1]]
country: [["Italy","Spain","Slovenia"]]

We can do the same with a polars dataframe:

>>> import polars as pl
>>> from datetime import datetime
>>> arr = [datetime(2023, 5, 20, 10, 0),
...        datetime(2023, 5, 20, 11, 0),
...        datetime(2023, 5, 20, 13, 30)]
>>> df = pl.DataFrame({
...          'Talk': ['About Polars','Intro into PyArrow','Coding in Rust'],
...          'Time': arr,
...      })
>>> df
shape: (3, 2)
┌────────────────────┬─────────────────────┐
│ Talk               ┆ Time                │
│ ---                ┆ ---                 │
│ str                ┆ datetime[μs]        │
╞════════════════════╪═════════════════════╡
│ About Polars       ┆ 2023-05-20 10:00:00 │
│ Intro into PyArrow ┆ 2023-05-20 11:00:00 │
│ Coding in Rust     ┆ 2023-05-20 13:30:00 │
└────────────────────┴─────────────────────┘
>>> from_dataframe(df)
pyarrow.Table
Talk: large_string
Time: timestamp[us]
----
Talk: [["About Polars","Intro into PyArrow","Coding in Rust"]]
Time: [[2023-05-20 10:00:00.000000,2023-05-20 11:00:00.000000,2023-05-20 13:30:00.000000]]