pyarrow.Table¶

class pyarrow.Table¶

Bases: pyarrow.lib._PandasConvertible

A collection of top-level named, equal length Arrow arrays.

Warning

Do not call this class’s constructor directly, use one of the from_* methods instead.

__init__(*args, **kwargs)¶: Initialize self. See help(type(self)) for accurate signature.

Methods

`__init__`(args, *kwargs)	Initialize self.
`add_column`(self, int i, field_, column)	Add column to Table at position.
`append_column`(self, field_, column)	Append column at end of columns.
`cast`(self, Schema target_schema, bool safe=True)	Cast table values to another schema.
`column`(self, i)	Select a column by its column name, or numeric index.
`combine_chunks`(self, MemoryPool memory_pool=None)	Make a new table by combining the chunks this table has.
`drop`(self, columns)	Drop one or more columns and return a new table.
`drop_null`(self)	Remove missing values from a Table.
`equals`(self, Table other, …)	Check if contents of two tables are equal.
`field`(self, i)	Select a schema field by its column name or numeric index.
`filter`(self, mask[, null_selection_behavior])	Select records from a Table.
`flatten`(self, MemoryPool memory_pool=None)	Flatten this Table.
`from_arrays`(arrays[, names, schema, metadata])	Construct a Table from Arrow arrays.
`from_batches`(batches, Schema schema=None)	Construct a Table from a sequence or iterator of Arrow RecordBatches.
`from_pandas`(type cls, df, Schema schema=None)	Convert pandas.DataFrame to an Arrow Table.
`from_pydict`(mapping[, schema, metadata])	Construct a Table from Arrow arrays or columns.
`itercolumns`(self)	Iterator over all columns in their numerical order.
`remove_column`(self, int i)	Create new Table with the indicated column removed.
`rename_columns`(self, names)	Create new table with columns renamed to provided names.
`replace_schema_metadata`(self[, metadata])	Create shallow copy of table by replacing schema key-value metadata with the indicated new metadata (which may be None), which deletes any existing metadata.
`select`(self, columns)	Select columns of the Table.
`set_column`(self, int i, field_, column)	Replace column in Table at position.
`slice`(self[, offset, length])	Compute zero-copy slice of this Table.
`take`(self, indices)	Select records from a Table.
`to_batches`(self[, max_chunksize])	Convert Table to list of (contiguous) RecordBatch objects.
`to_pandas`(self[, memory_pool, categories, …])	Convert to a pandas-compatible NumPy array or DataFrame, as appropriate
`to_pydict`(self)	Convert the Table to a dict or OrderedDict.
`to_string`(self, *[, show_metadata, preview_cols])	Return human-readable string representation of Table.
`unify_dictionaries`(self, …)	Unify dictionaries across all chunks.
`validate`(self, *[, full])	Perform validation checks.

Attributes

`column_names`	Names of the table’s columns.
`columns`	List of all columns in numerical order.
`nbytes`	Total number of bytes consumed by the elements of the table.
`num_columns`	Number of columns in this table.
`num_rows`	Number of rows in this table.
`schema`	Schema of the table and its columns.
`shape`	(#rows, #columns).

add_column(self, int i, field_, column)¶

Add column to Table at position.

A new table is returned with the column added, the original table object is left unchanged.

Parameters

i (int) – Index to place the column at.
field (str or Field) – If a string is passed then the type is deduced from the column data.
column (Array, list of Array, or values coercible to arrays) – Column data.

Returns

Table – New table with the passed column added.

append_column(self, field_, column)¶

Append column at end of columns.

Parameters

field (str or Field) – If a string is passed then the type is deduced from the column data.
column (Array, list of Array, or values coercible to arrays) – Column data.

Returns

Table – New table with the passed column added.

cast(self, Schema target_schema, bool safe=True)¶

Cast table values to another schema.

Parameters

target_schema (Schema) – Schema to cast to, the names and order of fields must match.
safe (bool, default True) – Check for overflows or other unsafe conversions.

Returns

Table

column(self, i)¶

Select a column by its column name, or numeric index.

Parameters: i (int or string) – The index or name of the column to retrieve.
Returns: ChunkedArray

column_names¶

Names of the table’s columns.

Returns: list of str

columns¶

List of all columns in numerical order.

Returns: list of ChunkedArray

combine_chunks(self, MemoryPool memory_pool=None)¶

Make a new table by combining the chunks this table has.

All the underlying chunks in the ChunkedArray of each column are concatenated into zero or one chunk.

Parameters: memory_pool (MemoryPool, default None) – For memory allocations, if required, otherwise use default pool.
Returns: Table

drop(self, columns)¶

Drop one or more columns and return a new table.

Parameters: columns (list of str) – List of field names referencing existing columns.
Raises: KeyError – If any of the passed columns name are not existing.
Returns: Table – New table without the columns.

drop_null(self)¶: Remove missing values from a Table. See pyarrow.compute.drop_null() for full usage.

equals(self, Table other, bool check_metadata=False)¶

Check if contents of two tables are equal.

Parameters

other (pyarrow.Table) – Table to compare against.
check_metadata (bool, default False) – Whether schema metadata equality should be checked as well.

Returns

bool

field(self, i)¶

Select a schema field by its column name or numeric index.

Parameters: i (int or string) – The index or name of the field to retrieve.
Returns: Field

filter(self, mask, null_selection_behavior='drop')¶: Select records from a Table. See pyarrow.compute.filter() for full usage.

flatten(self, MemoryPool memory_pool=None)¶

Flatten this Table.

Each column with a struct type is flattened into one column per struct field. Other columns are left unchanged.

Parameters: memory_pool (MemoryPool, default None) – For memory allocations, if required, otherwise use default pool
Returns: Table

static from_arrays(arrays, names=None, schema=None, metadata=None)¶

Construct a Table from Arrow arrays.

Parameters

arrays (list of pyarrow.Array or pyarrow.ChunkedArray) – Equal-length arrays that should form the table.
names (list of str, optional) – Names for the table columns. If not passed, schema must be passed.
schema (Schema, default None) – Schema for the created table. If not passed, names must be passed.
metadata (dict or Mapping, default None) – Optional metadata for the schema (if inferred).

Returns

Table

static from_batches(batches, Schema schema=None)¶

Construct a Table from a sequence or iterator of Arrow RecordBatches.

Parameters

batches (sequence or iterator of RecordBatch) – Sequence of RecordBatch to be converted, all schemas must be equal.
schema (Schema, default None) – If not passed, will be inferred from the first RecordBatch.

Returns

Table

from_pandas(type cls, df, Schema schema=None, preserve_index=None, nthreads=None, columns=None, bool safe=True)¶

Convert pandas.DataFrame to an Arrow Table.

The column types in the resulting Arrow Table are inferred from the dtypes of the pandas.Series in the DataFrame. In the case of non-object Series, the NumPy dtype is translated to its Arrow equivalent. In the case of object, we need to guess the datatype by looking at the Python objects in this Series.

Be aware that Series of the object dtype don’t carry enough information to always lead to a meaningful Arrow type. In the case that we cannot infer a type, e.g. because the DataFrame is of length 0 or the Series only contains None/nan objects, the type is set to null. This behavior can be avoided by constructing an explicit schema and passing it to this function.

Parameters

df (pandas.DataFrame) –
schema (pyarrow.Schema, optional) – The expected schema of the Arrow Table. This can be used to indicate the type of columns if we cannot infer it automatically. If passed, the output will have exactly this schema. Columns specified in the schema that are not found in the DataFrame columns or its index will raise an error. Additional columns or index levels in the DataFrame which are not specified in the schema will be ignored.
preserve_index (bool, optional) – Whether to store the index as an additional column in the resulting Table. The default of None will store the index as a column, except for RangeIndex which is stored as metadata only. Use preserve_index=True to force it to be stored as a column.
nthreads (int, default None (may use up to system CPU count threads)) – If greater than 1, convert columns to Arrow in parallel using indicated number of threads.
columns (list, optional) – List of column to be converted. If None, use all columns.
safe (bool, default True) – Check for overflows or other unsafe conversions.

Returns

Table

Examples

>>> import pandas as pd
>>> import pyarrow as pa
>>> df = pd.DataFrame({
    ...     'int': [1, 2],
    ...     'str': ['a', 'b']
    ... })
>>> pa.Table.from_pandas(df)
<pyarrow.lib.Table object at 0x7f05d1fb1b40>

static from_pydict(mapping, schema=None, metadata=None)¶

Construct a Table from Arrow arrays or columns.

Parameters

mapping (dict or Mapping) – A mapping of strings to Arrays or Python lists.
schema (Schema, default None) – If not passed, will be inferred from the Mapping values.
metadata (dict or Mapping, default None) – Optional metadata for the schema (if inferred).

Returns

Table

itercolumns(self)¶

Iterator over all columns in their numerical order.

Yields: ChunkedArray

nbytes¶

Total number of bytes consumed by the elements of the table.

Returns: int

num_columns¶

Number of columns in this table.

Returns: int

num_rows¶

Number of rows in this table.

Due to the definition of a table, all columns have the same number of rows.

Returns: int

remove_column(self, int i)¶

Create new Table with the indicated column removed.

Parameters: i (int) – Index of column to remove.
Returns: Table – New table without the column.

rename_columns(self, names)¶

Create new table with columns renamed to provided names.

Parameters: names (list of str) – List of new column names.
Returns: Table

replace_schema_metadata(self, metadata=None)¶

Create shallow copy of table by replacing schema key-value metadata with the indicated new metadata (which may be None), which deletes any existing metadata.

Parameters: metadata (dict, default None) –
Returns: Table

schema¶

Schema of the table and its columns.

Returns: Schema

select(self, columns)¶

Select columns of the Table.

Returns a new Table with the specified columns, and metadata preserved.

Parameters: columns (list-like) – The column names or integer indices to select.
Returns: Table

set_column(self, int i, field_, column)¶

Replace column in Table at position.

Parameters

i (int) – Index to place the column at.
field (str or Field) – If a string is passed then the type is deduced from the column data.
column (Array, list of Array, or values coercible to arrays) – Column data.

Returns

Table – New table with the passed column set.

shape¶

(#rows, #columns).

Returns: (int, int) – Number of rows and number of columns.
Type: Dimensions of the table

slice(self, offset=0, length=None)¶

Compute zero-copy slice of this Table.

Parameters

offset (int, default 0) – Offset from start of table to slice.
length (int, default None) – Length of slice (default is until end of table starting from offset).

Returns

Table

take(self, indices)¶: Select records from a Table. See pyarrow.compute.take() for full usage.

to_batches(self, max_chunksize=None, **kwargs)¶

Convert Table to list of (contiguous) RecordBatch objects.

Parameters: max_chunksize (int, default None) – Maximum size for RecordBatch chunks. Individual chunks may be smaller depending on the chunk layout of individual columns.
Returns: list of RecordBatch

to_pandas(self, memory_pool=None, categories=None, bool strings_to_categorical=False, bool zero_copy_only=False, bool integer_object_nulls=False, bool date_as_object=True, bool timestamp_as_object=False, bool use_threads=True, bool deduplicate_objects=True, bool ignore_metadata=False, bool safe=True, bool split_blocks=False, bool self_destruct=False, types_mapper=None)¶

Convert to a pandas-compatible NumPy array or DataFrame, as appropriate

Parameters

memory_pool (MemoryPool, default None) – Arrow MemoryPool to use for allocations. Uses the default memory pool is not passed.
strings_to_categorical (bool, default False) – Encode string (UTF8) and binary types to pandas.Categorical.
categories (list, default empty) – List of fields that should be returned as pandas.Categorical. Only applies to table-like data structures.
zero_copy_only (bool, default False) – Raise an ArrowException if this function call would require copying the underlying data.
integer_object_nulls (bool, default False) – Cast integers with nulls to objects
date_as_object (bool, default True) – Cast dates to objects. If False, convert to datetime64[ns] dtype.
timestamp_as_object (bool, default False) – Cast non-nanosecond timestamps (np.datetime64) to objects. This is useful if you have timestamps that don’t fit in the normal date range of nanosecond timestamps (1678 CE-2262 CE). If False, all timestamps are converted to datetime64[ns] dtype.
use_threads (bool, default True) – Whether to parallelize the conversion using multiple threads.
deduplicate_objects (bool, default False) – Do not create multiple copies Python objects when created, to save on memory use. Conversion will be slower.
ignore_metadata (bool, default False) – If True, do not use the ‘pandas’ metadata to reconstruct the DataFrame index, if present
safe (bool, default True) – For certain data types, a cast is needed in order to store the data in a pandas DataFrame or Series (e.g. timestamps are always stored as nanoseconds in pandas). This option controls whether it is a safe cast or not.
split_blocks (bool, default False) – If True, generate one internal “block” for each column when creating a pandas.DataFrame from a RecordBatch or Table. While this can temporarily reduce memory note that various pandas operations can trigger “consolidation” which may balloon memory use.
self_destruct (bool, default False) –
EXPERIMENTAL: If True, attempt to deallocate the originating Arrow memory while converting the Arrow object to pandas. If you use the object after calling to_pandas with this option it will crash your program.

Note that you may not see always memory usage improvements. For example, if multiple columns share an underlying allocation, memory can’t be freed until all columns are converted.
types_mapper (function, default None) – A function mapping a pyarrow DataType to a pandas ExtensionDtype. This can be used to override the default pandas type for conversion of built-in pyarrow types or in absence of pandas_metadata in the Table schema. The function receives a pyarrow DataType and is expected to return a pandas ExtensionDtype or None if the default conversion should be used for that type. If you have a dictionary mapping, you can pass dict.get as function.

Returns

pandas.Series or pandas.DataFrame depending on type of object

to_pydict(self)¶

Convert the Table to a dict or OrderedDict.

Returns: dict

to_string(self, *, show_metadata=False, preview_cols=0)¶

Return human-readable string representation of Table.

Parameters

show_metadata (bool, default True) – Display Field-level and Schema-level KeyValueMetadata.
preview_cols (int, default 0) – Display values of the columns for the first N columns.

Returns

str

unify_dictionaries(self, MemoryPool memory_pool=None)¶

Unify dictionaries across all chunks.

This method returns an equivalent table, but where all chunks of each column share the same dictionary values. Dictionary indices are transposed accordingly.

Columns without dictionaries are returned unchanged.

Parameters: memory_pool (MemoryPool, default None) – For memory allocations, if required, otherwise use default pool
Returns: Table

validate(self, *, full=False)¶

Perform validation checks. An exception is raised if validation fails.

By default only cheap validation checks are run. Pass full=True for thorough validation checks (potentially O(n)).

Parameters: full (bool, default False) – If True, run expensive checks, otherwise cheap checks only.
Raises: ArrowInvalid –

pyarrow.RecordBatch

pyarrow.Tensor