pyarrow.Table#

class pyarrow.Table#

Bases: _Tabular

A collection of top-level named, equal length Arrow arrays.

Warning

Do not call this class’s constructor directly, use one of the from_* methods instead.

Examples

>>> import pyarrow as pa
>>> n_legs = pa.array([2, 4, 5, 100])
>>> animals = pa.array(["Flamingo", "Horse", "Brittle stars", "Centipede"])
>>> names = ["n_legs", "animals"]

Construct a Table from arrays:

>>> pa.Table.from_arrays([n_legs, animals], names=names)
pyarrow.Table
n_legs: int64
animals: string
----
n_legs: [[2,4,5,100]]
animals: [["Flamingo","Horse","Brittle stars","Centipede"]]

Construct a Table from a RecordBatch:

>>> batch = pa.record_batch([n_legs, animals], names=names)
>>> pa.Table.from_batches([batch])
pyarrow.Table
n_legs: int64
animals: string
----
n_legs: [[2,4,5,100]]
animals: [["Flamingo","Horse","Brittle stars","Centipede"]]

Construct a Table from pandas DataFrame:

>>> import pandas as pd
>>> df = pd.DataFrame({'year': [2020, 2022, 2019, 2021],
...                    'n_legs': [2, 4, 5, 100],
...                    'animals': ["Flamingo", "Horse", "Brittle stars", "Centipede"]})
>>> pa.Table.from_pandas(df)
pyarrow.Table
year: int64
n_legs: int64
animals: string
----
year: [[2020,2022,2019,2021]]
n_legs: [[2,4,5,100]]
animals: [["Flamingo","Horse","Brittle stars","Centipede"]]

Construct a Table from a dictionary of arrays:

>>> pydict = {'n_legs': n_legs, 'animals': animals}
>>> pa.Table.from_pydict(pydict)
pyarrow.Table
n_legs: int64
animals: string
----
n_legs: [[2,4,5,100]]
animals: [["Flamingo","Horse","Brittle stars","Centipede"]]
>>> pa.Table.from_pydict(pydict).schema
n_legs: int64
animals: string

Construct a Table from a dictionary of arrays with metadata:

>>> my_metadata={"n_legs": "Number of legs per animal"}
>>> pa.Table.from_pydict(pydict, metadata=my_metadata).schema
n_legs: int64
animals: string
-- schema metadata --
n_legs: 'Number of legs per animal'

Construct a Table from a list of rows:

>>> pylist = [{'n_legs': 2, 'animals': 'Flamingo'}, {'year': 2021, 'animals': 'Centipede'}]
>>> pa.Table.from_pylist(pylist)
pyarrow.Table
n_legs: int64
animals: string
----
n_legs: [[2,null]]
animals: [["Flamingo","Centipede"]]

Construct a Table from a list of rows with pyarrow schema:

>>> my_schema = pa.schema([
...     pa.field('year', pa.int64()),
...     pa.field('n_legs', pa.int64()),
...     pa.field('animals', pa.string())],
...     metadata={"year": "Year of entry"})
>>> pa.Table.from_pylist(pylist, schema=my_schema).schema
year: int64
n_legs: int64
animals: string
-- schema metadata --
year: 'Year of entry'

Construct a Table with pyarrow.table():

>>> pa.table([n_legs, animals], names=names)
pyarrow.Table
n_legs: int64
animals: string
----
n_legs: [[2,4,5,100]]
animals: [["Flamingo","Horse","Brittle stars","Centipede"]]

__init__(*args, **kwargs)#

Methods

`__init__`(args, *kwargs)
`add_column`(self, int i, field_, column)	Add column to Table at position.
`append_column`(self, field_, column)	Append column at end of columns.
`cast`(self, Schema target_schema[, safe, options])	Cast table values to another schema.
`column`(self, i)	Select single column from Table or RecordBatch.
`combine_chunks`(self, MemoryPool memory_pool=None)	Make a new table by combining the chunks this table has.
`drop`(self, columns)	Drop one or more columns and return a new table.
`drop_columns`(self, columns)	Drop one or more columns and return a new Table or RecordBatch.
`drop_null`(self)	Remove rows that contain missing values from a Table or RecordBatch.
`equals`(self, Table other, ...)	Check if contents of two tables are equal.
`field`(self, i)	Select a schema field by its column name or numeric index.
`filter`(self, mask[, null_selection_behavior])	Select rows from the table or record batch based on a boolean mask.
`flatten`(self, MemoryPool memory_pool=None)	Flatten this Table.
`from_arrays`(arrays[, names, schema, metadata])	Construct a Table from Arrow arrays.
`from_batches`(batches, Schema schema=None)	Construct a Table from a sequence or iterator of Arrow RecordBatches.
`from_pandas`(cls, df, Schema schema=None[, ...])	Convert pandas.DataFrame to an Arrow Table.
`from_pydict`(cls, mapping[, schema, metadata])	Construct a Table or RecordBatch from Arrow arrays or columns.
`from_pylist`(cls, mapping[, schema, metadata])	Construct a Table or RecordBatch from list of rows / dictionaries.
`from_struct_array`(struct_array)	Construct a Table from a StructArray.
`get_total_buffer_size`(self)	The sum of bytes in each buffer referenced by the table.
`group_by`(self, keys[, use_threads])	Declare a grouping over the columns of the table.
`itercolumns`(self)	Iterator over all columns in their numerical order.
`join`(self, right_table, keys[, right_keys, ...])	Perform a join between this table and another one.
`join_asof`(self, right_table, on, by, tolerance)	Perform an asof join between this table and another one.
`remove_column`(self, int i)	Create new Table with the indicated column removed.
`rename_columns`(self, names)	Create new table with columns renamed to provided names.
`replace_schema_metadata`(self[, metadata])	Create shallow copy of table by replacing schema key-value metadata with the indicated new metadata (which may be None), which deletes any existing metadata.
`select`(self, columns)	Select columns of the Table.
`set_column`(self, int i, field_, column)	Replace column in Table at position.
`slice`(self[, offset, length])	Compute zero-copy slice of this Table.
`sort_by`(self, sorting, **kwargs)	Sort the Table or RecordBatch by one or multiple columns.
`take`(self, indices)	Select rows from a Table or RecordBatch.
`to_batches`(self[, max_chunksize])	Convert Table to a list of RecordBatch objects.
`to_pandas`(self[, memory_pool, categories, ...])	Convert to a pandas-compatible NumPy array or DataFrame, as appropriate
`to_pydict`(self)	Convert the Table or RecordBatch to a dict or OrderedDict.
`to_pylist`(self)	Convert the Table or RecordBatch to a list of rows / dictionaries.
`to_reader`(self[, max_chunksize])	Convert the Table to a RecordBatchReader.
`to_string`(self, *[, show_metadata, preview_cols])	Return human-readable string representation of Table or RecordBatch.
`to_struct_array`(self[, max_chunksize])	Convert to a chunked array of struct type.
`unify_dictionaries`(self, ...)	Unify dictionaries across all chunks.
`validate`(self, *[, full])	Perform validation checks.

Attributes

`column_names`	Names of the Table or RecordBatch columns.
`columns`	List of all columns in numerical order.
`is_cpu`	Whether all ChunkedArrays are CPU-accessible.
`nbytes`	Total number of bytes consumed by the elements of the table.
`num_columns`	Number of columns in this table.
`num_rows`	Number of rows in this table.
`schema`	Schema of the table and its columns.
`shape`	Dimensions of the table or record batch: (#rows, #columns).

__dataframe__(self, nan_as_null: bool = False, allow_copy: bool = True)#

Return the dataframe interchange object implementing the interchange protocol.

Parameters:

nan_as_nullbool, default False: Whether to tell the DataFrame to overwrite null values in the data with NaN (or NaT).
allow_copybool, default True: Whether to allow memory copying when exporting. If set to False it would cause non-zero-copy exports to fail.

Returns:

DataFrame interchange object: The object which consuming library can use to ingress the dataframe.

Notes

Details on the interchange protocol: https://data-apis.org/dataframe-protocol/latest/index.html nan_as_null currently has no effect; once support for nullable extension dtypes is added, this value should be propagated to columns.

add_column(self, int i, field_, column)#

Add column to Table at position.

A new table is returned with the column added, the original table object is left unchanged.

Parameters:

iint: Index to place the column at.
field_str or Field: If a string is passed then the type is deduced from the column data.
columnArray, list of Array, or values coercible to arrays: Column data.

Returns:

Table: New table with the passed column added.

Examples

>>> import pyarrow as pa
>>> import pandas as pd
>>> df = pd.DataFrame({'n_legs': [2, 4, 5, 100],
...                    'animals': ["Flamingo", "Horse", "Brittle stars", "Centipede"]})
>>> table = pa.Table.from_pandas(df)

Add column:

>>> year = [2021, 2022, 2019, 2021]
>>> table.add_column(0,"year", [year])
pyarrow.Table
year: int64
n_legs: int64
animals: string
----
year: [[2021,2022,2019,2021]]
n_legs: [[2,4,5,100]]
animals: [["Flamingo","Horse","Brittle stars","Centipede"]]

Original table is left unchanged:

>>> table
pyarrow.Table
n_legs: int64
animals: string
----
n_legs: [[2,4,5,100]]
animals: [["Flamingo","Horse","Brittle stars","Centipede"]]

append_column(self, field_, column)#

Append column at end of columns.

Parameters:

field_str or Field: If a string is passed then the type is deduced from the column data.
columnArray or value coercible to array: Column data.

Returns:

Table or RecordBatch: New table or record batch with the passed column added.

Examples

>>> import pyarrow as pa
>>> import pandas as pd
>>> df = pd.DataFrame({'n_legs': [2, 4, 5, 100],
...                    'animals': ["Flamingo", "Horse", "Brittle stars", "Centipede"]})
>>> table = pa.Table.from_pandas(df)

Append column at the end:

>>> year = [2021, 2022, 2019, 2021]
>>> table.append_column('year', [year])
pyarrow.Table
n_legs: int64
animals: string
year: int64
----
n_legs: [[2,4,5,100]]
animals: [["Flamingo","Horse","Brittle stars","Centipede"]]
year: [[2021,2022,2019,2021]]

cast(self, Schema target_schema, safe=None, options=None)#

Cast table values to another schema.

Parameters:

target_schemaSchema: Schema to cast to, the names and order of fields must match.
safebool, default True: Check for overflows or other unsafe conversions.
optionsCastOptions, default None: Additional checks pass by CastOptions

Returns:

Table

Examples

>>> import pyarrow as pa
>>> import pandas as pd
>>> df = pd.DataFrame({'n_legs': [2, 4, 5, 100],
...                    'animals': ["Flamingo", "Horse", "Brittle stars", "Centipede"]})
>>> table = pa.Table.from_pandas(df)
>>> table.schema
n_legs: int64
animals: string
-- schema metadata --
pandas: '{"index_columns": [{"kind": "range", "name": null, "start": 0, ...

Define new schema and cast table values:

>>> my_schema = pa.schema([
...     pa.field('n_legs', pa.duration('s')),
...     pa.field('animals', pa.string())]
...     )
>>> table.cast(target_schema=my_schema)
pyarrow.Table
n_legs: duration[s]
animals: string
----
n_legs: [[2,4,5,100]]
animals: [["Flamingo","Horse","Brittle stars","Centipede"]]

column(self, i)#

Select single column from Table or RecordBatch.

Parameters:

iint or str: The index or name of the column to retrieve.

Returns:

columnArray (for RecordBatch) or ChunkedArray (for Table)

Examples

Table (works similarly for RecordBatch)

>>> import pyarrow as pa
>>> import pandas as pd
>>> df = pd.DataFrame({'n_legs': [2, 4, 5, 100],
...                    'animals': ["Flamingo", "Horse", "Brittle stars", "Centipede"]})
>>> table = pa.Table.from_pandas(df)

Select a column by numeric index:

>>> table.column(0)
<pyarrow.lib.ChunkedArray object at ...>
[
  [
    2,
    4,
    5,
    100
  ]
]

Select a column by its name:

>>> table.column("animals")
<pyarrow.lib.ChunkedArray object at ...>
[
  [
    "Flamingo",
    "Horse",
    "Brittle stars",
    "Centipede"
  ]
]

column_names#

Names of the Table or RecordBatch columns.

Returns:

list of str

Examples

Table (works similarly for RecordBatch)

>>> import pyarrow as pa
>>> table = pa.Table.from_arrays([[2, 4, 5, 100],
...                               ["Flamingo", "Horse", "Brittle stars", "Centipede"]],
...                               names=['n_legs', 'animals'])
>>> table.column_names
['n_legs', 'animals']

columns#

List of all columns in numerical order.

Returns:

columnslist of Array (for RecordBatch) or list of ChunkedArray (for Table)

Examples

Table (works similarly for RecordBatch)

>>> import pyarrow as pa
>>> import pandas as pd
>>> df = pd.DataFrame({'n_legs': [None, 4, 5, None],
...                    'animals': ["Flamingo", "Horse", None, "Centipede"]})
>>> table = pa.Table.from_pandas(df)
>>> table.columns
[<pyarrow.lib.ChunkedArray object at ...>
[
  [
    null,
    4,
    5,
    null
  ]
], <pyarrow.lib.ChunkedArray object at ...>
[
  [
    "Flamingo",
    "Horse",
    null,
    "Centipede"
  ]
]]

combine_chunks(self, MemoryPool memory_pool=None)#

Make a new table by combining the chunks this table has.

All the underlying chunks in the ChunkedArray of each column are concatenated into zero or one chunk.

Parameters:

memory_poolMemoryPool, default None: For memory allocations, if required, otherwise use default pool.

Returns:

Table

Examples

>>> import pyarrow as pa
>>> n_legs = pa.chunked_array([[2, 2, 4], [4, 5, 100]])
>>> animals = pa.chunked_array([["Flamingo", "Parrot", "Dog"], ["Horse", "Brittle stars", "Centipede"]])
>>> names = ["n_legs", "animals"]
>>> table = pa.table([n_legs, animals], names=names)
>>> table
pyarrow.Table
n_legs: int64
animals: string
----
n_legs: [[2,2,4],[4,5,100]]
animals: [["Flamingo","Parrot","Dog"],["Horse","Brittle stars","Centipede"]]
>>> table.combine_chunks()
pyarrow.Table
n_legs: int64
animals: string
----
n_legs: [[2,2,4,4,5,100]]
animals: [["Flamingo","Parrot","Dog","Horse","Brittle stars","Centipede"]]

drop(self, columns)#

Drop one or more columns and return a new table.

Alias of Table.drop_columns, but kept for backwards compatibility.

Parameters:

columnsstr or list[str]: Field name(s) referencing existing column(s).

Returns:

Table: New table without the column(s).

drop_columns(self, columns)#

Drop one or more columns and return a new Table or RecordBatch.

Parameters:

columnsstr or list[str]: Field name(s) referencing existing column(s).

Returns:

Table or RecordBatch: A tabular object without the column(s).

Raises:

KeyError: If any of the passed column names do not exist.

Examples

Table (works similarly for RecordBatch)

>>> import pyarrow as pa
>>> import pandas as pd
>>> df = pd.DataFrame({'n_legs': [2, 4, 5, 100],
...                    'animals': ["Flamingo", "Horse", "Brittle stars", "Centipede"]})
>>> table = pa.Table.from_pandas(df)

Drop one column:

>>> table.drop_columns("animals")
pyarrow.Table
n_legs: int64
----
n_legs: [[2,4,5,100]]

Drop one or more columns:

>>> table.drop_columns(["n_legs", "animals"])
pyarrow.Table
...
----

drop_null(self)#

Remove rows that contain missing values from a Table or RecordBatch.

See pyarrow.compute.drop_null() for full usage.

Returns:

Table or RecordBatch: A tabular object with the same schema, with rows containing no missing values.

Examples

Table (works similarly for RecordBatch)

>>> import pyarrow as pa
>>> import pandas as pd
>>> df = pd.DataFrame({'year': [None, 2022, 2019, 2021],
...                   'n_legs': [2, 4, 5, 100],
...                   'animals': ["Flamingo", "Horse", None, "Centipede"]})
>>> table = pa.Table.from_pandas(df)
>>> table.drop_null()
pyarrow.Table
year: double
n_legs: int64
animals: string
----
year: [[2022,2021]]
n_legs: [[4,100]]
animals: [["Horse","Centipede"]]

equals(self, Table other, bool check_metadata=False)#

Check if contents of two tables are equal.

Parameters:

otherpyarrow.Table: Table to compare against.
check_metadatabool, default False: Whether schema metadata equality should be checked as well.

Returns:

bool

Examples

>>> import pyarrow as pa
>>> n_legs = pa.array([2, 2, 4, 4, 5, 100])
>>> animals = pa.array(["Flamingo", "Parrot", "Dog", "Horse", "Brittle stars", "Centipede"])
>>> names=["n_legs", "animals"]
>>> table = pa.Table.from_arrays([n_legs, animals], names=names)
>>> table_0 = pa.Table.from_arrays([])
>>> table_1 = pa.Table.from_arrays([n_legs, animals],
...                                 names=names,
...                                 metadata={"n_legs": "Number of legs per animal"})
>>> table.equals(table)
True
>>> table.equals(table_0)
False
>>> table.equals(table_1)
True
>>> table.equals(table_1, check_metadata=True)
False

field(self, i)#

Select a schema field by its column name or numeric index.

Parameters:

iint or str: The index or name of the field to retrieve.

Returns:

Field

Examples

Table (works similarly for RecordBatch)

>>> import pyarrow as pa
>>> import pandas as pd
>>> df = pd.DataFrame({'n_legs': [2, 4, 5, 100],
...                    'animals': ["Flamingo", "Horse", "Brittle stars", "Centipede"]})
>>> table = pa.Table.from_pandas(df)
>>> table.field(0)
pyarrow.Field<n_legs: int64>
>>> table.field(1)
pyarrow.Field<animals: string>

filter(self, mask, null_selection_behavior='drop')#

Select rows from the table or record batch based on a boolean mask.

The Table can be filtered based on a mask, which will be passed to pyarrow.compute.filter() to perform the filtering, or it can be filtered through a boolean Expression

Parameters:

maskArray or array-like or Expression: The boolean mask or the Expression to filter the table with.
null_selection_behaviorstr, default “drop”: How nulls in the mask should be handled, does nothing if an Expression is used.

Returns:

filteredTable or RecordBatch: A tabular object of the same schema, with only the rows selected by applied filtering

Examples

Using a Table (works similarly for RecordBatch):

>>> import pyarrow as pa
>>> table = pa.table({'year': [2020, 2022, 2019, 2021],
...                   'n_legs': [2, 4, 5, 100],
...                   'animals': ["Flamingo", "Horse", "Brittle stars", "Centipede"]})

Define an expression and select rows:

>>> import pyarrow.compute as pc
>>> expr = pc.field("year") <= 2020
>>> table.filter(expr)
pyarrow.Table
year: int64
n_legs: int64
animals: string
----
year: [[2020,2019]]
n_legs: [[2,5]]
animals: [["Flamingo","Brittle stars"]]

Define a mask and select rows:

>>> mask=[True, True, False, None]
>>> table.filter(mask)
pyarrow.Table
year: int64
n_legs: int64
animals: string
----
year: [[2020,2022]]
n_legs: [[2,4]]
animals: [["Flamingo","Horse"]]
>>> table.filter(mask, null_selection_behavior='emit_null')
pyarrow.Table
year: int64
n_legs: int64
animals: string
----
year: [[2020,2022,null]]
n_legs: [[2,4,null]]
animals: [["Flamingo","Horse",null]]

flatten(self, MemoryPool memory_pool=None)#

Flatten this Table.

Each column with a struct type is flattened into one column per struct field. Other columns are left unchanged.

Parameters:

memory_poolMemoryPool, default None: For memory allocations, if required, otherwise use default pool

Returns:

Table

Examples

>>> import pyarrow as pa
>>> struct = pa.array([{'n_legs': 2, 'animals': 'Parrot'},
...                    {'year': 2022, 'n_legs': 4}])
>>> month = pa.array([4, 6])
>>> table = pa.Table.from_arrays([struct,month],
...                              names = ["a", "month"])
>>> table
pyarrow.Table
a: struct<animals: string, n_legs: int64, year: int64>
  child 0, animals: string
  child 1, n_legs: int64
  child 2, year: int64
month: int64
----
a: [
  -- is_valid: all not null
  -- child 0 type: string
["Parrot",null]
  -- child 1 type: int64
[2,4]
  -- child 2 type: int64
[null,2022]]
month: [[4,6]]

Flatten the columns with struct field:

>>> table.flatten()
pyarrow.Table
a.animals: string
a.n_legs: int64
a.year: int64
month: int64
----
a.animals: [["Parrot",null]]
a.n_legs: [[2,4]]
a.year: [[null,2022]]
month: [[4,6]]

static from_arrays(arrays, names=None, schema=None, metadata=None)#

Construct a Table from Arrow arrays.

Parameters:

arrayslist of pyarrow.Array or pyarrow.ChunkedArray: Equal-length arrays that should form the table.
nameslist of str, optional: Names for the table columns. If not passed, schema must be passed.
schemaSchema, default None: Schema for the created table. If not passed, names must be passed.
metadatadict or Mapping, default None: Optional metadata for the schema (if inferred).

Returns:

Table

Examples

>>> import pyarrow as pa
>>> n_legs = pa.array([2, 4, 5, 100])
>>> animals = pa.array(["Flamingo", "Horse", "Brittle stars", "Centipede"])
>>> names = ["n_legs", "animals"]

Construct a Table from arrays:

>>> pa.Table.from_arrays([n_legs, animals], names=names)
pyarrow.Table
n_legs: int64
animals: string
----
n_legs: [[2,4,5,100]]
animals: [["Flamingo","Horse","Brittle stars","Centipede"]]

Construct a Table from arrays with metadata:

>>> my_metadata={"n_legs": "Number of legs per animal"}
>>> pa.Table.from_arrays([n_legs, animals],
...                       names=names,
...                       metadata=my_metadata)
pyarrow.Table
n_legs: int64
animals: string
----
n_legs: [[2,4,5,100]]
animals: [["Flamingo","Horse","Brittle stars","Centipede"]]
>>> pa.Table.from_arrays([n_legs, animals],
...                       names=names,
...                       metadata=my_metadata).schema
n_legs: int64
animals: string
-- schema metadata --
n_legs: 'Number of legs per animal'

Construct a Table from arrays with pyarrow schema:

>>> my_schema = pa.schema([
...     pa.field('n_legs', pa.int64()),
...     pa.field('animals', pa.string())],
...     metadata={"animals": "Name of the animal species"})
>>> pa.Table.from_arrays([n_legs, animals],
...                       schema=my_schema)
pyarrow.Table
n_legs: int64
animals: string
----
n_legs: [[2,4,5,100]]
animals: [["Flamingo","Horse","Brittle stars","Centipede"]]
>>> pa.Table.from_arrays([n_legs, animals],
...                       schema=my_schema).schema
n_legs: int64
animals: string
-- schema metadata --
animals: 'Name of the animal species'

static from_batches(batches, Schema schema=None)#

Construct a Table from a sequence or iterator of Arrow RecordBatches.

Parameters:

batchessequence or iterator of RecordBatch: Sequence of RecordBatch to be converted, all schemas must be equal.
schemaSchema, default None: If not passed, will be inferred from the first RecordBatch.

Returns:

Table

Examples

>>> import pyarrow as pa
>>> n_legs = pa.array([2, 4, 5, 100])
>>> animals = pa.array(["Flamingo", "Horse", "Brittle stars", "Centipede"])
>>> names = ["n_legs", "animals"]
>>> batch = pa.record_batch([n_legs, animals], names=names)
>>> batch.to_pandas()
   n_legs        animals
0       2       Flamingo
1       4          Horse
2       5  Brittle stars
3     100      Centipede

Construct a Table from a RecordBatch:

>>> pa.Table.from_batches([batch])
pyarrow.Table
n_legs: int64
animals: string
----
n_legs: [[2,4,5,100]]
animals: [["Flamingo","Horse","Brittle stars","Centipede"]]

Construct a Table from a sequence of RecordBatches:

>>> pa.Table.from_batches([batch, batch])
pyarrow.Table
n_legs: int64
animals: string
----
n_legs: [[2,4,5,100],[2,4,5,100]]
animals: [["Flamingo","Horse","Brittle stars","Centipede"],["Flamingo","Horse","Brittle stars","Centipede"]]

classmethod from_pandas(cls, df, Schema schema=None, preserve_index=None, nthreads=None, columns=None, bool safe=True)#

Convert pandas.DataFrame to an Arrow Table.

The column types in the resulting Arrow Table are inferred from the dtypes of the pandas.Series in the DataFrame. In the case of non-object Series, the NumPy dtype is translated to its Arrow equivalent. In the case of object, we need to guess the datatype by looking at the Python objects in this Series.

Be aware that Series of the object dtype don’t carry enough information to always lead to a meaningful Arrow type. In the case that we cannot infer a type, e.g. because the DataFrame is of length 0 or the Series only contains None/nan objects, the type is set to null. This behavior can be avoided by constructing an explicit schema and passing it to this function.

Parameters:

dfpandas.DataFrame
schemapyarrow.Schema, optional: The expected schema of the Arrow Table. This can be used to indicate the type of columns if we cannot infer it automatically. If passed, the output will have exactly this schema. Columns specified in the schema that are not found in the DataFrame columns or its index will raise an error. Additional columns or index levels in the DataFrame which are not specified in the schema will be ignored.
preserve_indexbool, optional: Whether to store the index as an additional column in the resulting Table. The default of None will store the index as a column, except for RangeIndex which is stored as metadata only. Use preserve_index=True to force it to be stored as a column.
nthreadsint, default None: If greater than 1, convert columns to Arrow in parallel using indicated number of threads. By default, this follows pyarrow.cpu_count() (may use up to system CPU count threads).
columnslist, optional: List of column to be converted. If None, use all columns.
safebool, default True: Check for overflows or other unsafe conversions.

Returns:

Table

Examples

>>> import pyarrow as pa
>>> import pandas as pd
>>> df = pd.DataFrame({'n_legs': [2, 4, 5, 100],
...                    'animals': ["Flamingo", "Horse", "Brittle stars", "Centipede"]})
>>> pa.Table.from_pandas(df)
pyarrow.Table
n_legs: int64
animals: string
----
n_legs: [[2,4,5,100]]
animals: [["Flamingo","Horse","Brittle stars","Centipede"]]

classmethod from_pydict(cls, mapping, schema=None, metadata=None)#

Construct a Table or RecordBatch from Arrow arrays or columns.

Parameters:

mappingdict or Mapping: A mapping of strings to Arrays or Python lists.
schemaSchema, default None: If not passed, will be inferred from the Mapping values.
metadatadict or Mapping, default None: Optional metadata for the schema (if inferred).

Returns:

Table or RecordBatch

Examples

Table (works similarly for RecordBatch)

>>> import pyarrow as pa
>>> n_legs = pa.array([2, 4, 5, 100])
>>> animals = pa.array(["Flamingo", "Horse", "Brittle stars", "Centipede"])
>>> pydict = {'n_legs': n_legs, 'animals': animals}

Construct a Table from a dictionary of arrays:

>>> pa.Table.from_pydict(pydict)
pyarrow.Table
n_legs: int64
animals: string
----
n_legs: [[2,4,5,100]]
animals: [["Flamingo","Horse","Brittle stars","Centipede"]]
>>> pa.Table.from_pydict(pydict).schema
n_legs: int64
animals: string

Construct a Table from a dictionary of arrays with metadata:

>>> my_metadata={"n_legs": "Number of legs per animal"}
>>> pa.Table.from_pydict(pydict, metadata=my_metadata).schema
n_legs: int64
animals: string
-- schema metadata --
n_legs: 'Number of legs per animal'

Construct a Table from a dictionary of arrays with pyarrow schema:

>>> my_schema = pa.schema([
...     pa.field('n_legs', pa.int64()),
...     pa.field('animals', pa.string())],
...     metadata={"n_legs": "Number of legs per animal"})
>>> pa.Table.from_pydict(pydict, schema=my_schema).schema
n_legs: int64
animals: string
-- schema metadata --
n_legs: 'Number of legs per animal'

classmethod from_pylist(cls, mapping, schema=None, metadata=None)#

Construct a Table or RecordBatch from list of rows / dictionaries.

Parameters:

mappinglist of dicts of rows: A mapping of strings to row values.
schemaSchema, default None: If not passed, will be inferred from the first row of the mapping values.
metadatadict or Mapping, default None: Optional metadata for the schema (if inferred).

Returns:

Table or RecordBatch

Examples

Table (works similarly for RecordBatch)

>>> import pyarrow as pa
>>> pylist = [{'n_legs': 2, 'animals': 'Flamingo'},
...           {'n_legs': 4, 'animals': 'Dog'}]

Construct a Table from a list of rows:

>>> pa.Table.from_pylist(pylist)
pyarrow.Table
n_legs: int64
animals: string
----
n_legs: [[2,4]]
animals: [["Flamingo","Dog"]]

Construct a Table from a list of rows with metadata:

>>> my_metadata={"n_legs": "Number of legs per animal"}
>>> pa.Table.from_pylist(pylist, metadata=my_metadata).schema
n_legs: int64
animals: string
-- schema metadata --
n_legs: 'Number of legs per animal'

Construct a Table from a list of rows with pyarrow schema:

>>> my_schema = pa.schema([
...     pa.field('n_legs', pa.int64()),
...     pa.field('animals', pa.string())],
...     metadata={"n_legs": "Number of legs per animal"})
>>> pa.Table.from_pylist(pylist, schema=my_schema).schema
n_legs: int64
animals: string
-- schema metadata --
n_legs: 'Number of legs per animal'

static from_struct_array(struct_array)#

Construct a Table from a StructArray.

Each field in the StructArray will become a column in the resulting Table.

Parameters:

struct_arrayStructArray or ChunkedArray: Array to construct the table from.

Returns:

pyarrow.Table

Examples

>>> import pyarrow as pa
>>> struct = pa.array([{'n_legs': 2, 'animals': 'Parrot'},
...                    {'year': 2022, 'n_legs': 4}])
>>> pa.Table.from_struct_array(struct).to_pandas()
  animals  n_legs    year
0  Parrot       2     NaN
1    None       4  2022.0

get_total_buffer_size(self)#

The sum of bytes in each buffer referenced by the table.

An array may only reference a portion of a buffer. This method will overestimate in this case and return the byte size of the entire buffer.

If a buffer is referenced multiple times then it will only be counted once.

Examples

>>> import pyarrow as pa
>>> import pandas as pd
>>> df = pd.DataFrame({'n_legs': [None, 4, 5, None],
...                    'animals': ["Flamingo", "Horse", None, "Centipede"]})
>>> table = pa.Table.from_pandas(df)
>>> table.get_total_buffer_size()
76

group_by(self, keys, use_threads=True)#

Declare a grouping over the columns of the table.

Resulting grouping can then be used to perform aggregations with a subsequent aggregate() method.

Parameters:

keysstr or list[str]: Name of the columns that should be used as the grouping key.
use_threadsbool, default True: Whether to use multithreading or not. When set to True (the default), no stable ordering of the output is guaranteed.

Returns:

TableGroupBy

See also

TableGroupBy.aggregate

Examples

>>> import pandas as pd
>>> import pyarrow as pa
>>> df = pd.DataFrame({'year': [2020, 2022, 2021, 2022, 2019, 2021],
...                    'n_legs': [2, 2, 4, 4, 5, 100],
...                    'animal': ["Flamingo", "Parrot", "Dog", "Horse",
...                    "Brittle stars", "Centipede"]})
>>> table = pa.Table.from_pandas(df)
>>> table.group_by('year').aggregate([('n_legs', 'sum')])
pyarrow.Table
year: int64
n_legs_sum: int64
----
year: [[2020,2022,2021,2019]]
n_legs_sum: [[2,6,104,5]]

is_cpu#: Whether all ChunkedArrays are CPU-accessible.

itercolumns(self)#

Iterator over all columns in their numerical order.

Yields:

Array (for RecordBatch) or ChunkedArray (for Table)

Examples

Table (works similarly for RecordBatch)

>>> import pyarrow as pa
>>> import pandas as pd
>>> df = pd.DataFrame({'n_legs': [None, 4, 5, None],
...                    'animals': ["Flamingo", "Horse", None, "Centipede"]})
>>> table = pa.Table.from_pandas(df)
>>> for i in table.itercolumns():
...     print(i.null_count)
...
2
1

join(self, right_table, keys, right_keys=None, join_type='left outer', left_suffix=None, right_suffix=None, coalesce_keys=True, use_threads=True)#

Perform a join between this table and another one.

Result of the join will be a new Table, where further operations can be applied.

Parameters:

right_tableTable: The table to join to the current one, acting as the right table in the join operation.
keysstr or list[str]: The columns from current table that should be used as keys of the join operation left side.
right_keysstr or list[str], default None: The columns from the right_table that should be used as keys on the join operation right side. When None use the same key names as the left table.
join_typestr, default “left outer”: The kind of join that should be performed, one of (“left semi”, “right semi”, “left anti”, “right anti”, “inner”, “left outer”, “right outer”, “full outer”)
left_suffixstr, default None: Which suffix to add to left column names. This prevents confusion when the columns in left and right tables have colliding names.
right_suffixstr, default None: Which suffix to add to the right column names. This prevents confusion when the columns in left and right tables have colliding names.
coalesce_keysbool, default True: If the duplicated keys should be omitted from one of the sides in the join result.
use_threadsbool, default True: Whether to use multithreading or not.

Returns:

Table

Examples

>>> import pandas as pd
>>> import pyarrow as pa
>>> df1 = pd.DataFrame({'id': [1, 2, 3],
...                     'year': [2020, 2022, 2019]})
>>> df2 = pd.DataFrame({'id': [3, 4],
...                     'n_legs': [5, 100],
...                     'animal': ["Brittle stars", "Centipede"]})
>>> t1 = pa.Table.from_pandas(df1)
>>> t2 = pa.Table.from_pandas(df2)

Left outer join:

>>> t1.join(t2, 'id').combine_chunks().sort_by('year')
pyarrow.Table
id: int64
year: int64
n_legs: int64
animal: string
----
id: [[3,1,2]]
year: [[2019,2020,2022]]
n_legs: [[5,null,null]]
animal: [["Brittle stars",null,null]]

Full outer join:

>>> t1.join(t2, 'id', join_type="full outer").combine_chunks().sort_by('year')
pyarrow.Table
id: int64
year: int64
n_legs: int64
animal: string
----
id: [[3,1,2,4]]
year: [[2019,2020,2022,null]]
n_legs: [[5,null,null,100]]
animal: [["Brittle stars",null,null,"Centipede"]]

Right outer join:

>>> t1.join(t2, 'id', join_type="right outer").combine_chunks().sort_by('year')
pyarrow.Table
year: int64
id: int64
n_legs: int64
animal: string
----
year: [[2019,null]]
id: [[3,4]]
n_legs: [[5,100]]
animal: [["Brittle stars","Centipede"]]

Right anti join

>>> t1.join(t2, 'id', join_type="right anti")
pyarrow.Table
id: int64
n_legs: int64
animal: string
----
id: [[4]]
n_legs: [[100]]
animal: [["Centipede"]]

join_asof(self, right_table, on, by, tolerance, right_on=None, right_by=None)#

Perform an asof join between this table and another one.

This is similar to a left-join except that we match on nearest key rather than equal keys. Both tables must be sorted by the key. This type of join is most useful for time series data that are not perfectly aligned.

Optionally match on equivalent keys with “by” before searching with “on”.

Result of the join will be a new Table, where further operations can be applied.

Parameters:

right_tableTable

The table to join to the current one, acting as the right table in the join operation.

onstr

The column from current table that should be used as the “on” key of the join operation left side.

An inexact match is used on the “on” key, i.e. a row is considered a match if and only if left_on - tolerance <= right_on <= left_on.

The input dataset must be sorted by the “on” key. Must be a single field of a common type.

Currently, the “on” key must be an integer, date, or timestamp type.

bystr or list[str]

The columns from current table that should be used as the keys of the join operation left side. The join operation is then done only for the matches in these columns.

toleranceint

The tolerance for inexact “on” key matching. A right row is considered a match with the left row right.on - left.on <= tolerance. The tolerance may be:

negative, in which case a past-as-of-join occurs;
or positive, in which case a future-as-of-join occurs;
or zero, in which case an exact-as-of-join occurs.

The tolerance is interpreted in the same units as the “on” key.

right_onstr or list[str], default None

The columns from the right_table that should be used as the on key on the join operation right side. When None use the same key name as the left table.

right_bystr or list[str], default None

The columns from the right_table that should be used as keys on the join operation right side. When None use the same key names as the left table.

Returns:

Table

nbytes#

Total number of bytes consumed by the elements of the table.

In other words, the sum of bytes from all buffer ranges referenced.

Unlike get_total_buffer_size this method will account for array offsets.

If buffers are shared between arrays then the shared portion will only be counted multiple times.

The dictionary of dictionary arrays will always be counted in their entirety even if the array only references a portion of the dictionary.

Examples

>>> import pyarrow as pa
>>> import pandas as pd
>>> df = pd.DataFrame({'n_legs': [None, 4, 5, None],
...                    'animals': ["Flamingo", "Horse", None, "Centipede"]})
>>> table = pa.Table.from_pandas(df)
>>> table.nbytes
72

num_columns#

Number of columns in this table.

Returns:

int

Examples

>>> import pyarrow as pa
>>> import pandas as pd
>>> df = pd.DataFrame({'n_legs': [None, 4, 5, None],
...                    'animals': ["Flamingo", "Horse", None, "Centipede"]})
>>> table = pa.Table.from_pandas(df)
>>> table.num_columns
2

num_rows#

Number of rows in this table.

Due to the definition of a table, all columns have the same number of rows.

Returns:

int

Examples

>>> import pyarrow as pa
>>> import pandas as pd
>>> df = pd.DataFrame({'n_legs': [None, 4, 5, None],
...                    'animals': ["Flamingo", "Horse", None, "Centipede"]})
>>> table = pa.Table.from_pandas(df)
>>> table.num_rows
4

remove_column(self, int i)#

Create new Table with the indicated column removed.

Parameters:

iint: Index of column to remove.

Returns:

Table: New table without the column.

Examples

>>> import pyarrow as pa
>>> import pandas as pd
>>> df = pd.DataFrame({'n_legs': [2, 4, 5, 100],
...                    'animals': ["Flamingo", "Horse", "Brittle stars", "Centipede"]})
>>> table = pa.Table.from_pandas(df)
>>> table.remove_column(1)
pyarrow.Table
n_legs: int64
----
n_legs: [[2,4,5,100]]

rename_columns(self, names)#

Create new table with columns renamed to provided names.

Parameters:

nameslist[str] or dict[str, str]

List of new column names or mapping of old column names to new column names.

If a mapping of old to new column names is passed, then all columns which are found to match a provided old column name will be renamed to the new column name. If any column names are not found in the mapping, a KeyError will be raised.

Returns:

Table

Raises:

KeyError: If any of the column names passed in the names mapping do not exist.

Examples

>>> import pyarrow as pa
>>> import pandas as pd
>>> df = pd.DataFrame({'n_legs': [2, 4, 5, 100],
...                    'animals': ["Flamingo", "Horse", "Brittle stars", "Centipede"]})
>>> table = pa.Table.from_pandas(df)
>>> new_names = ["n", "name"]
>>> table.rename_columns(new_names)
pyarrow.Table
n: int64
name: string
----
n: [[2,4,5,100]]
name: [["Flamingo","Horse","Brittle stars","Centipede"]]
>>> new_names = {"n_legs": "n", "animals": "name"}
>>> table.rename_columns(new_names)
pyarrow.Table
n: int64
name: string
----
n: [[2,4,5,100]]
name: [["Flamingo","Horse","Brittle stars","Centipede"]]

replace_schema_metadata(self, metadata=None)#

Create shallow copy of table by replacing schema key-value metadata with the indicated new metadata (which may be None), which deletes any existing metadata.

Parameters:

metadatadict, default None

Returns:

Table

Examples

>>> import pyarrow as pa
>>> import pandas as pd
>>> df = pd.DataFrame({'year': [2020, 2022, 2019, 2021],
...                    'n_legs': [2, 4, 5, 100],
...                    'animals': ["Flamingo", "Horse", "Brittle stars", "Centipede"]})
>>> table = pa.Table.from_pandas(df)

Constructing a Table with pyarrow schema and metadata:

>>> my_schema = pa.schema([
...     pa.field('n_legs', pa.int64()),
...     pa.field('animals', pa.string())],
...     metadata={"n_legs": "Number of legs per animal"})
>>> table= pa.table(df, my_schema)
>>> table.schema
n_legs: int64
animals: string
-- schema metadata --
n_legs: 'Number of legs per animal'
pandas: ...

Create a shallow copy of a Table with deleted schema metadata:

>>> table.replace_schema_metadata().schema
n_legs: int64
animals: string

Create a shallow copy of a Table with new schema metadata:

>>> metadata={"animals": "Which animal"}
>>> table.replace_schema_metadata(metadata = metadata).schema
n_legs: int64
animals: string
-- schema metadata --
animals: 'Which animal'

schema#

Schema of the table and its columns.

Returns:

Schema

Examples

>>> import pyarrow as pa
>>> import pandas as pd
>>> df = pd.DataFrame({'n_legs': [2, 4, 5, 100],
...                    'animals': ["Flamingo", "Horse", "Brittle stars", "Centipede"]})
>>> table = pa.Table.from_pandas(df)
>>> table.schema
n_legs: int64
animals: string
-- schema metadata --
pandas: '{"index_columns": [{"kind": "range", "name": null, "start": 0, "' ...

select(self, columns)#

Select columns of the Table.

Returns a new Table with the specified columns, and metadata preserved.

Parameters:

columnslist-like: The column names or integer indices to select.

Returns:

Table

Examples

>>> import pyarrow as pa
>>> import pandas as pd
>>> df = pd.DataFrame({'year': [2020, 2022, 2019, 2021],
...                    'n_legs': [2, 4, 5, 100],
...                    'animals': ["Flamingo", "Horse", "Brittle stars", "Centipede"]})
>>> table = pa.Table.from_pandas(df)
>>> table.select([0,1])
pyarrow.Table
year: int64
n_legs: int64
----
year: [[2020,2022,2019,2021]]
n_legs: [[2,4,5,100]]
>>> table.select(["year"])
pyarrow.Table
year: int64
----
year: [[2020,2022,2019,2021]]

set_column(self, int i, field_, column)#

Replace column in Table at position.

Parameters:

iint: Index to place the column at.
field_str or Field: If a string is passed then the type is deduced from the column data.
columnArray, list of Array, or values coercible to arrays: Column data.

Returns:

Table: New table with the passed column set.

Examples

>>> import pyarrow as pa
>>> import pandas as pd
>>> df = pd.DataFrame({'n_legs': [2, 4, 5, 100],
...                    'animals': ["Flamingo", "Horse", "Brittle stars", "Centipede"]})
>>> table = pa.Table.from_pandas(df)

Replace a column:

>>> year = [2021, 2022, 2019, 2021]
>>> table.set_column(1,'year', [year])
pyarrow.Table
n_legs: int64
year: int64
----
n_legs: [[2,4,5,100]]
year: [[2021,2022,2019,2021]]

shape#

Dimensions of the table or record batch: (#rows, #columns).

Returns:

(int, int): Number of rows and number of columns.

Examples

>>> import pyarrow as pa
>>> table = pa.table({'n_legs': [None, 4, 5, None],
...                   'animals': ["Flamingo", "Horse", None, "Centipede"]})
>>> table.shape
(4, 2)

slice(self, offset=0, length=None)#

Compute zero-copy slice of this Table.

Parameters:

offsetint, default 0: Offset from start of table to slice.
lengthint, default None: Length of slice (default is until end of table starting from offset).

Returns:

Table

Examples

>>> import pyarrow as pa
>>> import pandas as pd
>>> df = pd.DataFrame({'year': [2020, 2022, 2019, 2021],
...                    'n_legs': [2, 4, 5, 100],
...                    'animals': ["Flamingo", "Horse", "Brittle stars", "Centipede"]})
>>> table = pa.Table.from_pandas(df)
>>> table.slice(length=3)
pyarrow.Table
year: int64
n_legs: int64
animals: string
----
year: [[2020,2022,2019]]
n_legs: [[2,4,5]]
animals: [["Flamingo","Horse","Brittle stars"]]
>>> table.slice(offset=2)
pyarrow.Table
year: int64
n_legs: int64
animals: string
----
year: [[2019,2021]]
n_legs: [[5,100]]
animals: [["Brittle stars","Centipede"]]
>>> table.slice(offset=2, length=1)
pyarrow.Table
year: int64
n_legs: int64
animals: string
----
year: [[2019]]
n_legs: [[5]]
animals: [["Brittle stars"]]

sort_by(self, sorting, **kwargs)#

Sort the Table or RecordBatch by one or multiple columns.

Parameters:

sortingstr or list[tuple(name, order)]: Name of the column to use to sort (ascending), or a list of multiple sorting conditions where each entry is a tuple with column name and sorting order (“ascending” or “descending”)
**kwargsdict, optional: Additional sorting options. As allowed by SortOptions

Returns:

Table or RecordBatch: A new tabular object sorted according to the sort keys.

Examples

Table (works similarly for RecordBatch)

>>> import pandas as pd
>>> import pyarrow as pa
>>> df = pd.DataFrame({'year': [2020, 2022, 2021, 2022, 2019, 2021],
...                    'n_legs': [2, 2, 4, 4, 5, 100],
...                    'animal': ["Flamingo", "Parrot", "Dog", "Horse",
...                    "Brittle stars", "Centipede"]})
>>> table = pa.Table.from_pandas(df)
>>> table.sort_by('animal')
pyarrow.Table
year: int64
n_legs: int64
animal: string
----
year: [[2019,2021,2021,2020,2022,2022]]
n_legs: [[5,100,4,2,4,2]]
animal: [["Brittle stars","Centipede","Dog","Flamingo","Horse","Parrot"]]

take(self, indices)#

Select rows from a Table or RecordBatch.

See pyarrow.compute.take() for full usage.

Parameters:

indicesArray or array-like: The indices in the tabular object whose rows will be returned.

Returns:

Table or RecordBatch: A tabular object with the same schema, containing the taken rows.

Examples

Table (works similarly for RecordBatch)

>>> import pyarrow as pa
>>> import pandas as pd
>>> df = pd.DataFrame({'year': [2020, 2022, 2019, 2021],
...                    'n_legs': [2, 4, 5, 100],
...                    'animals': ["Flamingo", "Horse", "Brittle stars", "Centipede"]})
>>> table = pa.Table.from_pandas(df)
>>> table.take([1,3])
pyarrow.Table
year: int64
n_legs: int64
animals: string
----
year: [[2022,2021]]
n_legs: [[4,100]]
animals: [["Horse","Centipede"]]

to_batches(self, max_chunksize=None)#

Convert Table to a list of RecordBatch objects.

Note that this method is zero-copy, it merely exposes the same data under a different API.

Parameters:

max_chunksizeint, default None: Maximum number of rows for each RecordBatch chunk. Individual chunks may be smaller depending on the chunk layout of individual columns.

Returns:

list[RecordBatch]

Examples

>>> import pyarrow as pa
>>> import pandas as pd
>>> df = pd.DataFrame({'n_legs': [2, 4, 5, 100],
...                    'animals': ["Flamingo", "Horse", "Brittle stars", "Centipede"]})
>>> table = pa.Table.from_pandas(df)

Convert a Table to a RecordBatch:

>>> table.to_batches()[0].to_pandas()
   n_legs        animals
0       2       Flamingo
1       4          Horse
2       5  Brittle stars
3     100      Centipede

Convert a Table to a list of RecordBatches:

>>> table.to_batches(max_chunksize=2)[0].to_pandas()
   n_legs   animals
0       2  Flamingo
1       4     Horse
>>> table.to_batches(max_chunksize=2)[1].to_pandas()
   n_legs        animals
0       5  Brittle stars
1     100      Centipede

to_pandas(self, memory_pool=None, categories=None, bool strings_to_categorical=False, bool zero_copy_only=False, bool integer_object_nulls=False, bool date_as_object=True, bool timestamp_as_object=False, bool use_threads=True, bool deduplicate_objects=True, bool ignore_metadata=False, bool safe=True, bool split_blocks=False, bool self_destruct=False, unicode maps_as_pydicts=None, types_mapper=None, bool coerce_temporal_nanoseconds=False)#

Convert to a pandas-compatible NumPy array or DataFrame, as appropriate

Parameters:

memory_poolMemoryPool, default None

Arrow MemoryPool to use for allocations. Uses the default memory pool if not passed.

categorieslist, default empty

List of fields that should be returned as pandas.Categorical. Only applies to table-like data structures.

strings_to_categoricalbool, default False

Encode string (UTF8) and binary types to pandas.Categorical.

zero_copy_onlybool, default False

Raise an ArrowException if this function call would require copying the underlying data.

integer_object_nullsbool, default False

Cast integers with nulls to objects

date_as_objectbool, default True

Cast dates to objects. If False, convert to datetime64 dtype with the equivalent time unit (if supported). Note: in pandas version < 2.0, only datetime64[ns] conversion is supported.

timestamp_as_objectbool, default False

Cast non-nanosecond timestamps (np.datetime64) to objects. This is useful in pandas version 1.x if you have timestamps that don’t fit in the normal date range of nanosecond timestamps (1678 CE-2262 CE). Non-nanosecond timestamps are supported in pandas version 2.0. If False, all timestamps are converted to datetime64 dtype.

use_threadsbool, default True

Whether to parallelize the conversion using multiple threads.

deduplicate_objectsbool, default True

Do not create multiple copies Python objects when created, to save on memory use. Conversion will be slower.

ignore_metadatabool, default False

If True, do not use the ‘pandas’ metadata to reconstruct the DataFrame index, if present

safebool, default True

For certain data types, a cast is needed in order to store the data in a pandas DataFrame or Series (e.g. timestamps are always stored as nanoseconds in pandas). This option controls whether it is a safe cast or not.

split_blocksbool, default False

If True, generate one internal “block” for each column when creating a pandas.DataFrame from a RecordBatch or Table. While this can temporarily reduce memory note that various pandas operations can trigger “consolidation” which may balloon memory use.

self_destructbool, default False

EXPERIMENTAL: If True, attempt to deallocate the originating Arrow memory while converting the Arrow object to pandas. If you use the object after calling to_pandas with this option it will crash your program.

Note that you may not see always memory usage improvements. For example, if multiple columns share an underlying allocation, memory can’t be freed until all columns are converted.

maps_as_pydictsstr, optional, default None

Valid values are None, ‘lossy’, or ‘strict’. The default behavior (None), is to convert Arrow Map arrays to Python association lists (list-of-tuples) in the same order as the Arrow Map, as in [(key1, value1), (key2, value2), …].

If ‘lossy’ or ‘strict’, convert Arrow Map arrays to native Python dicts. This can change the ordering of (key, value) pairs, and will deduplicate multiple keys, resulting in a possible loss of data.

If ‘lossy’, this key deduplication results in a warning printed when detected. If ‘strict’, this instead results in an exception being raised when detected.

types_mapperfunction, default None

A function mapping a pyarrow DataType to a pandas ExtensionDtype. This can be used to override the default pandas type for conversion of built-in pyarrow types or in absence of pandas_metadata in the Table schema. The function receives a pyarrow DataType and is expected to return a pandas ExtensionDtype or None if the default conversion should be used for that type. If you have a dictionary mapping, you can pass dict.get as function.

coerce_temporal_nanosecondsbool, default False

Only applicable to pandas version >= 2.0. A legacy option to coerce date32, date64, duration, and timestamp time units to nanoseconds when converting to pandas. This is the default behavior in pandas version 1.x. Set this option to True if you’d like to use this coercion when using pandas version >= 2.0 for backwards compatibility (not recommended otherwise).

Returns:

pandas.Series or pandas.DataFrame depending on type of object

Examples

>>> import pyarrow as pa
>>> import pandas as pd

Convert a Table to pandas DataFrame:

>>> table = pa.table([
...    pa.array([2, 4, 5, 100]),
...    pa.array(["Flamingo", "Horse", "Brittle stars", "Centipede"])
...    ], names=['n_legs', 'animals'])
>>> table.to_pandas()
   n_legs        animals
0       2       Flamingo
1       4          Horse
2       5  Brittle stars
3     100      Centipede
>>> isinstance(table.to_pandas(), pd.DataFrame)
True

Convert a RecordBatch to pandas DataFrame:

>>> import pyarrow as pa
>>> n_legs = pa.array([2, 4, 5, 100])
>>> animals = pa.array(["Flamingo", "Horse", "Brittle stars", "Centipede"])
>>> batch = pa.record_batch([n_legs, animals],
...                         names=["n_legs", "animals"])
>>> batch
pyarrow.RecordBatch
n_legs: int64
animals: string
----
n_legs: [2,4,5,100]
animals: ["Flamingo","Horse","Brittle stars","Centipede"]
>>> batch.to_pandas()
   n_legs        animals
0       2       Flamingo
1       4          Horse
2       5  Brittle stars
3     100      Centipede
>>> isinstance(batch.to_pandas(), pd.DataFrame)
True

Convert a Chunked Array to pandas Series:

>>> import pyarrow as pa
>>> n_legs = pa.chunked_array([[2, 2, 4], [4, 5, 100]])
>>> n_legs.to_pandas()
0      2
1      2
2      4
3      4
4      5
5    100
dtype: int64
>>> isinstance(n_legs.to_pandas(), pd.Series)
True

to_pydict(self)#

Convert the Table or RecordBatch to a dict or OrderedDict.

Returns:

dict

Examples

Table (works similarly for RecordBatch)

>>> import pyarrow as pa
>>> n_legs = pa.array([2, 2, 4, 4, 5, 100])
>>> animals = pa.array(["Flamingo", "Parrot", "Dog", "Horse", "Brittle stars", "Centipede"])
>>> table = pa.Table.from_arrays([n_legs, animals], names=["n_legs", "animals"])
>>> table.to_pydict()
{'n_legs': [2, 2, 4, 4, 5, 100], 'animals': ['Flamingo', 'Parrot', ..., 'Centipede']}

to_pylist(self)#

Convert the Table or RecordBatch to a list of rows / dictionaries.

Returns:

list

Examples

Table (works similarly for RecordBatch)

>>> import pyarrow as pa
>>> data = [[2, 4, 5, 100],
...         ["Flamingo", "Horse", "Brittle stars", "Centipede"]]
>>> table = pa.table(data, names=["n_legs", "animals"])
>>> table.to_pylist()
[{'n_legs': 2, 'animals': 'Flamingo'}, {'n_legs': 4, 'animals': 'Horse'}, ...

to_reader(self, max_chunksize=None)#

Convert the Table to a RecordBatchReader.

Note that this method is zero-copy, it merely exposes the same data under a different API.

Parameters:

max_chunksizeint, default None: Maximum number of rows for each RecordBatch chunk. Individual chunks may be smaller depending on the chunk layout of individual columns.

Returns:

RecordBatchReader

Examples

>>> import pyarrow as pa
>>> import pandas as pd
>>> df = pd.DataFrame({'n_legs': [2, 4, 5, 100],
...                    'animals': ["Flamingo", "Horse", "Brittle stars", "Centipede"]})
>>> table = pa.Table.from_pandas(df)

Convert a Table to a RecordBatchReader:

>>> table.to_reader()
<pyarrow.lib.RecordBatchReader object at ...>

>>> reader = table.to_reader()
>>> reader.schema
n_legs: int64
animals: string
-- schema metadata --
pandas: '{"index_columns": [{"kind": "range", "name": null, "start": 0, ...
>>> reader.read_all()
pyarrow.Table
n_legs: int64
animals: string
----
n_legs: [[2,4,5,100]]
animals: [["Flamingo","Horse","Brittle stars","Centipede"]]

to_string(self, *, show_metadata=False, preview_cols=0)#

Return human-readable string representation of Table or RecordBatch.

Parameters:

show_metadatabool, default False: Display Field-level and Schema-level KeyValueMetadata.
preview_colsint, default 0: Display values of the columns for the first N columns.

Returns:

str

to_struct_array(self, max_chunksize=None)#

Convert to a chunked array of struct type.

Parameters:

max_chunksizeint, default None: Maximum number of rows for ChunkedArray chunks. Individual chunks may be smaller depending on the chunk layout of individual columns.

Returns:

ChunkedArray

unify_dictionaries(self, MemoryPool memory_pool=None)#

Unify dictionaries across all chunks.

This method returns an equivalent table, but where all chunks of each column share the same dictionary values. Dictionary indices are transposed accordingly.

Columns without dictionaries are returned unchanged.

Parameters:

memory_poolMemoryPool, default None: For memory allocations, if required, otherwise use default pool

Returns:

Table

Examples

>>> import pyarrow as pa
>>> arr_1 = pa.array(["Flamingo", "Parrot", "Dog"]).dictionary_encode()
>>> arr_2 = pa.array(["Horse", "Brittle stars", "Centipede"]).dictionary_encode()
>>> c_arr = pa.chunked_array([arr_1, arr_2])
>>> table = pa.table([c_arr], names=["animals"])
>>> table
pyarrow.Table
animals: dictionary<values=string, indices=int32, ordered=0>
----
animals: [  -- dictionary:
["Flamingo","Parrot","Dog"]  -- indices:
[0,1,2],  -- dictionary:
["Horse","Brittle stars","Centipede"]  -- indices:
[0,1,2]]

Unify dictionaries across both chunks:

>>> table.unify_dictionaries()
pyarrow.Table
animals: dictionary<values=string, indices=int32, ordered=0>
----
animals: [  -- dictionary:
["Flamingo","Parrot","Dog","Horse","Brittle stars","Centipede"]  -- indices:
[0,1,2],  -- dictionary:
["Flamingo","Parrot","Dog","Horse","Brittle stars","Centipede"]  -- indices:
[3,4,5]]

validate(self, *, full=False)#

Perform validation checks. An exception is raised if validation fails.

By default only cheap validation checks are run. Pass full=True for thorough validation checks (potentially O(n)).

Parameters:

fullbool, default False: If True, run expensive checks, otherwise cheap checks only.

Raises:

ArrowInvalid