pyarrow.RecordBatch¶

class pyarrow.RecordBatch¶

Bases: _Tabular

Batch of rows of columns of equal length

Warning

Do not call this class’s constructor directly, use one of the RecordBatch.from_* functions instead.

Examples

>>> import pyarrow as pa
>>> n_legs = pa.array([2, 2, 4, 4, 5, 100])
>>> animals = pa.array(["Flamingo", "Parrot", "Dog", "Horse", "Brittle stars", "Centipede"])
>>> names = ["n_legs", "animals"]

Constructing a RecordBatch from arrays:

>>> pa.RecordBatch.from_arrays([n_legs, animals], names=names)
pyarrow.RecordBatch
n_legs: int64
animals: string
----
n_legs: [2,2,4,4,5,100]
animals: ["Flamingo","Parrot","Dog","Horse","Brittle stars","Centipede"]
>>> pa.RecordBatch.from_arrays([n_legs, animals], names=names).to_pandas()
   n_legs        animals
0       2       Flamingo
1       2         Parrot
2       4            Dog
3       4          Horse
4       5  Brittle stars
5     100      Centipede

Constructing a RecordBatch from pandas DataFrame:

>>> import pandas as pd
>>> df = pd.DataFrame({'year': [2020, 2022, 2021, 2022],
...                    'month': [3, 5, 7, 9],
...                    'day': [1, 5, 9, 13],
...                    'n_legs': [2, 4, 5, 100],
...                    'animals': ["Flamingo", "Horse", "Brittle stars", "Centipede"]})
>>> pa.RecordBatch.from_pandas(df)
pyarrow.RecordBatch
year: int64
month: int64
day: int64
n_legs: int64
animals: string
----
year: [2020,2022,2021,2022]
month: [3,5,7,9]
day: [1,5,9,13]
n_legs: [2,4,5,100]
animals: ["Flamingo","Horse","Brittle stars","Centipede"]
>>> pa.RecordBatch.from_pandas(df).to_pandas()
   year  month  day  n_legs        animals
0  2020      3    1       2       Flamingo
1  2022      5    5       4          Horse
2  2021      7    9       5  Brittle stars
3  2022      9   13     100      Centipede

Constructing a RecordBatch from pylist:

>>> pylist = [{'n_legs': 2, 'animals': 'Flamingo'},
...           {'n_legs': 4, 'animals': 'Dog'}]
>>> pa.RecordBatch.from_pylist(pylist).to_pandas()
   n_legs   animals
0       2  Flamingo
1       4       Dog

You can also construct a RecordBatch using pyarrow.record_batch():

>>> pa.record_batch([n_legs, animals], names=names).to_pandas()
   n_legs        animals
     2       Flamingo
     2         Parrot
     4            Dog
     4          Horse
     5  Brittle stars
   100      Centipede

>>> pa.record_batch(df)
pyarrow.RecordBatch
year: int64
month: int64
day: int64
n_legs: int64
animals: string
----
year: [2020,2022,2021,2022]
month: [3,5,7,9]
day: [1,5,9,13]
n_legs: [2,4,5,100]
animals: ["Flamingo","Horse","Brittle stars","Centipede"]

__init__(*args, **kwargs)¶

Methods

`__init__`(args, *kwargs)
`column`(self, i)	Select single column from Table or RecordBatch.
`drop_null`(self)	Remove rows that contain missing values from a Table or RecordBatch.
`equals`(self, other, bool check_metadata=False)	Check if contents of two record batches are equal.
`field`(self, i)	Select a schema field by its column name or numeric index.
`filter`(self, mask[, null_selection_behavior])	Select rows from the record batch.
`from_arrays`(list arrays[, names, schema, ...])	Construct a RecordBatch from multiple pyarrow.Arrays
`from_pandas`(type cls, df, Schema schema=None)	Convert pandas.DataFrame to an Arrow RecordBatch
`from_pydict`(type cls, mapping[, schema, ...])	Construct a Table or RecordBatch from Arrow arrays or columns.
`from_pylist`(type cls, mapping[, schema, ...])	Construct a Table or RecordBatch from list of rows / dictionaries.
`from_struct_array`(StructArray struct_array)	Construct a RecordBatch from a StructArray.
`get_total_buffer_size`(self)	The sum of bytes in each buffer referenced by the record batch
`itercolumns`(self)	Iterator over all columns in their numerical order.
`replace_schema_metadata`(self[, metadata])	Create shallow copy of record batch by replacing schema key-value metadata with the indicated new metadata (which may be None, which deletes any existing metadata
`select`(self, columns)	Select columns of the RecordBatch.
`serialize`(self[, memory_pool])	Write RecordBatch to Buffer as encapsulated IPC message, which does not include a Schema.
`slice`(self[, offset, length])	Compute zero-copy slice of this RecordBatch
`sort_by`(self, sorting, **kwargs)	Sort the Table or RecordBatch by one or multiple columns.
`take`(self, indices)	Select rows from a Table or RecordBatch.
`to_pandas`(self[, memory_pool, categories, ...])	Convert to a pandas-compatible NumPy array or DataFrame, as appropriate
`to_pydict`(self)	Convert the Table or RecordBatch to a dict or OrderedDict.
`to_pylist`(self)	Convert the Table or RecordBatch to a list of rows / dictionaries.
`to_string`(self, *[, show_metadata, preview_cols])	Return human-readable string representation of Table or RecordBatch.
`to_struct_array`(self)	Convert to a struct array.
`validate`(self, *[, full])	Perform validation checks.

Attributes

`column_names`	Names of the Table or RecordBatch columns.
`columns`	List of all columns in numerical order.
`nbytes`	Total number of bytes consumed by the elements of the record batch.
`num_columns`	Number of columns
`num_rows`	Number of rows
`schema`	Schema of the RecordBatch and its columns

__dataframe__(self, nan_as_null: bool = False, allow_copy: bool = True)¶

Return the dataframe interchange object implementing the interchange protocol.

Parameters:

nan_as_nullbool, default False: Whether to tell the DataFrame to overwrite null values in the data with NaN (or NaT).
allow_copybool, default True: Whether to allow memory copying when exporting. If set to False it would cause non-zero-copy exports to fail.

Returns:

DataFrame interchange object: The object which consuming library can use to ingress the dataframe.

Notes

Details on the interchange protocol: https://data-apis.org/dataframe-protocol/latest/index.html nan_as_null currently has no effect; once support for nullable extension dtypes is added, this value should be propagated to columns.

column(self, i)¶

Select single column from Table or RecordBatch.

Parameters:

iint or str: The index or name of the column to retrieve.

Returns:

columnArray (for RecordBatch) or ChunkedArray (for Table)

Examples

Table (works similarly for RecordBatch)

>>> import pyarrow as pa
>>> import pandas as pd
>>> df = pd.DataFrame({'n_legs': [2, 4, 5, 100],
...                    'animals': ["Flamingo", "Horse", "Brittle stars", "Centipede"]})
>>> table = pa.Table.from_pandas(df)

Select a column by numeric index:

>>> table.column(0)
<pyarrow.lib.ChunkedArray object at ...>
[
  [
    2,
    4,
    5,
    100
  ]
]

Select a column by its name:

>>> table.column("animals")
<pyarrow.lib.ChunkedArray object at ...>
[
  [
    "Flamingo",
    "Horse",
    "Brittle stars",
    "Centipede"
  ]
]

column_names¶

Names of the Table or RecordBatch columns.

Returns:

list of str

Examples

Table (works similarly for RecordBatch)

>>> import pyarrow as pa
>>> table = pa.Table.from_arrays([[2, 4, 5, 100],
...                               ["Flamingo", "Horse", "Brittle stars", "Centipede"]],
...                               names=['n_legs', 'animals'])
>>> table.column_names
['n_legs', 'animals']

columns¶

List of all columns in numerical order.

Returns:

columnslist of Array (for RecordBatch) or list of ChunkedArray (for Table)

Examples

Table (works similarly for RecordBatch)

>>> import pyarrow as pa
>>> import pandas as pd
>>> df = pd.DataFrame({'n_legs': [None, 4, 5, None],
...                    'animals': ["Flamingo", "Horse", None, "Centipede"]})
>>> table = pa.Table.from_pandas(df)
>>> table.columns
[<pyarrow.lib.ChunkedArray object at ...>
[
  [
    null,
    4,
    5,
    null
  ]
], <pyarrow.lib.ChunkedArray object at ...>
[
  [
    "Flamingo",
    "Horse",
    null,
    "Centipede"
  ]
]]

drop_null(self)¶

Remove rows that contain missing values from a Table or RecordBatch.

See pyarrow.compute.drop_null() for full usage.

Returns:

Table or RecordBatch: A tabular object with the same schema, with rows containing no missing values.

Examples

Table (works similarly for RecordBatch)

>>> import pyarrow as pa
>>> import pandas as pd
>>> df = pd.DataFrame({'year': [None, 2022, 2019, 2021],
...                   'n_legs': [2, 4, 5, 100],
...                   'animals': ["Flamingo", "Horse", None, "Centipede"]})
>>> table = pa.Table.from_pandas(df)
>>> table.drop_null()
pyarrow.Table
year: double
n_legs: int64
animals: string
----
year: [[2022,2021]]
n_legs: [[4,100]]
animals: [["Horse","Centipede"]]

equals(self, other, bool check_metadata=False)¶

Check if contents of two record batches are equal.

Parameters:

otherpyarrow.RecordBatch: RecordBatch to compare against.
check_metadatabool, default False: Whether schema metadata equality should be checked as well.

Returns:

are_equalbool

Examples

>>> import pyarrow as pa
>>> n_legs = pa.array([2, 2, 4, 4, 5, 100])
>>> animals = pa.array(["Flamingo", "Parrot", "Dog", "Horse", "Brittle stars", "Centipede"])
>>> batch = pa.RecordBatch.from_arrays([n_legs, animals],
...                                     names=["n_legs", "animals"])
>>> batch_0 = pa.record_batch([])
>>> batch_1 = pa.RecordBatch.from_arrays([n_legs, animals],
...                                       names=["n_legs", "animals"],
...                                       metadata={"n_legs": "Number of legs per animal"})
>>> batch.equals(batch)
True
>>> batch.equals(batch_0)
False
>>> batch.equals(batch_1)
True
>>> batch.equals(batch_1, check_metadata=True)
False

field(self, i)¶

Select a schema field by its column name or numeric index.

Parameters:

iint or str: The index or name of the field to retrieve.

Returns:

Field

Examples

Table (works similarly for RecordBatch)

>>> import pyarrow as pa
>>> import pandas as pd
>>> df = pd.DataFrame({'n_legs': [2, 4, 5, 100],
...                    'animals': ["Flamingo", "Horse", "Brittle stars", "Centipede"]})
>>> table = pa.Table.from_pandas(df)
>>> table.field(0)
pyarrow.Field<n_legs: int64>
>>> table.field(1)
pyarrow.Field<animals: string>

filter(self, mask, null_selection_behavior='drop')¶

Select rows from the record batch.

See pyarrow.compute.filter() for full usage.

Parameters:

maskArray or array-like: The boolean mask to filter the record batch with.
null_selection_behaviorstr, default “drop”: How nulls in the mask should be handled.

Returns:

filteredRecordBatch: A record batch of the same schema, with only the rows selected by the boolean mask.

Examples

>>> import pyarrow as pa
>>> n_legs = pa.array([2, 2, 4, 4, 5, 100])
>>> animals = pa.array(["Flamingo", "Parrot", "Dog", "Horse", "Brittle stars", "Centipede"])
>>> batch = pa.RecordBatch.from_arrays([n_legs, animals],
...                                     names=["n_legs", "animals"])
>>> batch.to_pandas()
   n_legs        animals
0       2       Flamingo
1       2         Parrot
2       4            Dog
3       4          Horse
4       5  Brittle stars
5     100      Centipede

Define a mask and select rows:

>>> mask=[True, True, False, True, False, None]
>>> batch.filter(mask).to_pandas()
   n_legs   animals
0       2  Flamingo
1       2    Parrot
2       4     Horse
>>> batch.filter(mask, null_selection_behavior='emit_null').to_pandas()
   n_legs   animals
0     2.0  Flamingo
1     2.0    Parrot
2     4.0     Horse
3     NaN      None

static from_arrays(list arrays, names=None, schema=None, metadata=None)¶

Construct a RecordBatch from multiple pyarrow.Arrays

Parameters:

arrayslist of pyarrow.Array: One for each field in RecordBatch
nameslist of str, optional: Names for the batch fields. If not passed, schema must be passed
schemaSchema, default None: Schema for the created batch. If not passed, names must be passed
metadatadict or Mapping, default None: Optional metadata for the schema (if inferred).

Returns:

pyarrow.RecordBatch

Examples

>>> import pyarrow as pa
>>> n_legs = pa.array([2, 2, 4, 4, 5, 100])
>>> animals = pa.array(["Flamingo", "Parrot", "Dog", "Horse", "Brittle stars", "Centipede"])
>>> names = ["n_legs", "animals"]

Construct a RecordBartch from pyarrow Arrays using names:

>>> pa.RecordBatch.from_arrays([n_legs, animals], names=names)
pyarrow.RecordBatch
n_legs: int64
animals: string
----
n_legs: [2,2,4,4,5,100]
animals: ["Flamingo","Parrot","Dog","Horse","Brittle stars","Centipede"]
>>> pa.RecordBatch.from_arrays([n_legs, animals], names=names).to_pandas()
   n_legs        animals
0       2       Flamingo
1       2         Parrot
2       4            Dog
3       4          Horse
4       5  Brittle stars
5     100      Centipede

Construct a RecordBartch from pyarrow Arrays using schema:

>>> my_schema = pa.schema([
...     pa.field('n_legs', pa.int64()),
...     pa.field('animals', pa.string())],
...     metadata={"n_legs": "Number of legs per animal"})
>>> pa.RecordBatch.from_arrays([n_legs, animals], schema=my_schema).to_pandas()
   n_legs        animals
0       2       Flamingo
1       2         Parrot
2       4            Dog
3       4          Horse
4       5  Brittle stars
5     100      Centipede
>>> pa.RecordBatch.from_arrays([n_legs, animals], schema=my_schema).schema
n_legs: int64
animals: string
-- schema metadata --
n_legs: 'Number of legs per animal'

from_pandas(type cls, df, Schema schema=None, preserve_index=None, nthreads=None, columns=None)¶

Convert pandas.DataFrame to an Arrow RecordBatch

Parameters:

dfpandas.DataFrame
schemapyarrow.Schema, optional: The expected schema of the RecordBatch. This can be used to indicate the type of columns if we cannot infer it automatically. If passed, the output will have exactly this schema. Columns specified in the schema that are not found in the DataFrame columns or its index will raise an error. Additional columns or index levels in the DataFrame which are not specified in the schema will be ignored.
preserve_indexbool, optional: Whether to store the index as an additional column in the resulting RecordBatch. The default of None will store the index as a column, except for RangeIndex which is stored as metadata only. Use preserve_index=True to force it to be stored as a column.
nthreadsint, default None: If greater than 1, convert columns to Arrow in parallel using indicated number of threads. By default, this follows pyarrow.cpu_count() (may use up to system CPU count threads).
columnslist, optional: List of column to be converted. If None, use all columns.

Returns:

pyarrow.RecordBatch

Examples

>>> import pandas as pd
>>> df = pd.DataFrame({'year': [2020, 2022, 2021, 2022],
...                    'month': [3, 5, 7, 9],
...                    'day': [1, 5, 9, 13],
...                    'n_legs': [2, 4, 5, 100],
...                    'animals': ["Flamingo", "Horse", "Brittle stars", "Centipede"]})

Convert pandas DataFrame to RecordBatch:

>>> import pyarrow as pa
>>> pa.RecordBatch.from_pandas(df)
pyarrow.RecordBatch
year: int64
month: int64
day: int64
n_legs: int64
animals: string
----
year: [2020,2022,2021,2022]
month: [3,5,7,9]
day: [1,5,9,13]
n_legs: [2,4,5,100]
animals: ["Flamingo","Horse","Brittle stars","Centipede"]

Convert pandas DataFrame to RecordBatch using schema:

>>> my_schema = pa.schema([
...     pa.field('n_legs', pa.int64()),
...     pa.field('animals', pa.string())],
...     metadata={"n_legs": "Number of legs per animal"})
>>> pa.RecordBatch.from_pandas(df, schema=my_schema)
pyarrow.RecordBatch
n_legs: int64
animals: string
----
n_legs: [2,4,5,100]
animals: ["Flamingo","Horse","Brittle stars","Centipede"]

Convert pandas DataFrame to RecordBatch specifying columns:

>>> pa.RecordBatch.from_pandas(df, columns=["n_legs"])
pyarrow.RecordBatch
n_legs: int64
----
n_legs: [2,4,5,100]

from_pydict(type cls, mapping, schema=None, metadata=None)¶

Construct a Table or RecordBatch from Arrow arrays or columns.

Parameters:

mappingdict or Mapping: A mapping of strings to Arrays or Python lists.
schemaSchema, default None: If not passed, will be inferred from the Mapping values.
metadatadict or Mapping, default None: Optional metadata for the schema (if inferred).

Returns:

Table or RecordBatch

Examples

Table (works similarly for RecordBatch)

>>> import pyarrow as pa
>>> n_legs = pa.array([2, 4, 5, 100])
>>> animals = pa.array(["Flamingo", "Horse", "Brittle stars", "Centipede"])
>>> pydict = {'n_legs': n_legs, 'animals': animals}

Construct a Table from a dictionary of arrays:

>>> pa.Table.from_pydict(pydict)
pyarrow.Table
n_legs: int64
animals: string
----
n_legs: [[2,4,5,100]]
animals: [["Flamingo","Horse","Brittle stars","Centipede"]]
>>> pa.Table.from_pydict(pydict).schema
n_legs: int64
animals: string

Construct a Table from a dictionary of arrays with metadata:

>>> my_metadata={"n_legs": "Number of legs per animal"}
>>> pa.Table.from_pydict(pydict, metadata=my_metadata).schema
n_legs: int64
animals: string
-- schema metadata --
n_legs: 'Number of legs per animal'

Construct a Table from a dictionary of arrays with pyarrow schema:

>>> my_schema = pa.schema([
...     pa.field('n_legs', pa.int64()),
...     pa.field('animals', pa.string())],
...     metadata={"n_legs": "Number of legs per animal"})
>>> pa.Table.from_pydict(pydict, schema=my_schema).schema
n_legs: int64
animals: string
-- schema metadata --
n_legs: 'Number of legs per animal'

from_pylist(type cls, mapping, schema=None, metadata=None)¶

Construct a Table or RecordBatch from list of rows / dictionaries.

Parameters:

mappinglist of dicts of rows: A mapping of strings to row values.
schemaSchema, default None: If not passed, will be inferred from the first row of the mapping values.
metadatadict or Mapping, default None: Optional metadata for the schema (if inferred).

Returns:

Table or RecordBatch

Examples

Table (works similarly for RecordBatch)

>>> import pyarrow as pa
>>> pylist = [{'n_legs': 2, 'animals': 'Flamingo'},
...           {'n_legs': 4, 'animals': 'Dog'}]

Construct a Table from a list of rows:

>>> pa.Table.from_pylist(pylist)
pyarrow.Table
n_legs: int64
animals: string
----
n_legs: [[2,4]]
animals: [["Flamingo","Dog"]]

Construct a Table from a list of rows with metadata:

>>> my_metadata={"n_legs": "Number of legs per animal"}
>>> pa.Table.from_pylist(pylist, metadata=my_metadata).schema
n_legs: int64
animals: string
-- schema metadata --
n_legs: 'Number of legs per animal'

Construct a Table from a list of rows with pyarrow schema:

>>> my_schema = pa.schema([
...     pa.field('n_legs', pa.int64()),
...     pa.field('animals', pa.string())],
...     metadata={"n_legs": "Number of legs per animal"})
>>> pa.Table.from_pylist(pylist, schema=my_schema).schema
n_legs: int64
animals: string
-- schema metadata --
n_legs: 'Number of legs per animal'

static from_struct_array(StructArray struct_array)¶

Construct a RecordBatch from a StructArray.

Each field in the StructArray will become a column in the resulting RecordBatch.

Parameters:

struct_arrayStructArray: Array to construct the record batch from.

Returns:

pyarrow.RecordBatch

Examples

>>> import pyarrow as pa
>>> struct = pa.array([{'n_legs': 2, 'animals': 'Parrot'},
...                    {'year': 2022, 'n_legs': 4}])
>>> pa.RecordBatch.from_struct_array(struct).to_pandas()
  animals  n_legs    year
0  Parrot       2     NaN
1    None       4  2022.0

get_total_buffer_size(self)¶

The sum of bytes in each buffer referenced by the record batch

An array may only reference a portion of a buffer. This method will overestimate in this case and return the byte size of the entire buffer.

If a buffer is referenced multiple times then it will only be counted once.

Examples

>>> import pyarrow as pa
>>> n_legs = pa.array([2, 2, 4, 4, 5, 100])
>>> animals = pa.array(["Flamingo", "Parrot", "Dog", "Horse", "Brittle stars", "Centipede"])
>>> batch = pa.RecordBatch.from_arrays([n_legs, animals],
...                                     names=["n_legs", "animals"])
>>> batch.get_total_buffer_size()
120

itercolumns(self)¶

Iterator over all columns in their numerical order.

Yields:

Array (for RecordBatch) or ChunkedArray (for Table)

Examples

Table (works similarly for RecordBatch)

>>> import pyarrow as pa
>>> import pandas as pd
>>> df = pd.DataFrame({'n_legs': [None, 4, 5, None],
...                    'animals': ["Flamingo", "Horse", None, "Centipede"]})
>>> table = pa.Table.from_pandas(df)
>>> for i in table.itercolumns():
...     print(i.null_count)
...
2
1

nbytes¶

Total number of bytes consumed by the elements of the record batch.

In other words, the sum of bytes from all buffer ranges referenced.

Unlike get_total_buffer_size this method will account for array offsets.

If buffers are shared between arrays then the shared portion will only be counted multiple times.

The dictionary of dictionary arrays will always be counted in their entirety even if the array only references a portion of the dictionary.

Examples

>>> import pyarrow as pa
>>> n_legs = pa.array([2, 2, 4, 4, 5, 100])
>>> animals = pa.array(["Flamingo", "Parrot", "Dog", "Horse", "Brittle stars", "Centipede"])
>>> batch = pa.RecordBatch.from_arrays([n_legs, animals],
...                                     names=["n_legs", "animals"])
>>> batch.nbytes
116

num_columns¶

Number of columns

Returns:

int

Examples

>>> import pyarrow as pa
>>> n_legs = pa.array([2, 2, 4, 4, 5, 100])
>>> animals = pa.array(["Flamingo", "Parrot", "Dog", "Horse", "Brittle stars", "Centipede"])
>>> batch = pa.RecordBatch.from_arrays([n_legs, animals],
...                                     names=["n_legs", "animals"])
>>> batch.num_columns
2

num_rows¶

Number of rows

Due to the definition of a RecordBatch, all columns have the same number of rows.

Returns:

int

Examples

>>> import pyarrow as pa
>>> n_legs = pa.array([2, 2, 4, 4, 5, 100])
>>> animals = pa.array(["Flamingo", "Parrot", "Dog", "Horse", "Brittle stars", "Centipede"])
>>> batch = pa.RecordBatch.from_arrays([n_legs, animals],
...                                     names=["n_legs", "animals"])
>>> batch.num_rows
6

replace_schema_metadata(self, metadata=None)¶

Create shallow copy of record batch by replacing schema key-value metadata with the indicated new metadata (which may be None, which deletes any existing metadata

Parameters:

metadatadict, default None

Returns:

shallow_copyRecordBatch

Examples

>>> import pyarrow as pa
>>> n_legs = pa.array([2, 2, 4, 4, 5, 100])

Constructing a RecordBatch with schema and metadata:

>>> my_schema = pa.schema([
...     pa.field('n_legs', pa.int64())],
...     metadata={"n_legs": "Number of legs per animal"})
>>> batch = pa.RecordBatch.from_arrays([n_legs], schema=my_schema)
>>> batch.schema
n_legs: int64
-- schema metadata --
n_legs: 'Number of legs per animal'

Shallow copy of a RecordBatch with deleted schema metadata:

>>> batch.replace_schema_metadata().schema
n_legs: int64

schema¶

Schema of the RecordBatch and its columns

Returns:

pyarrow.Schema

Examples

>>> import pyarrow as pa
>>> n_legs = pa.array([2, 2, 4, 4, 5, 100])
>>> animals = pa.array(["Flamingo", "Parrot", "Dog", "Horse", "Brittle stars", "Centipede"])
>>> batch = pa.RecordBatch.from_arrays([n_legs, animals],
...                                     names=["n_legs", "animals"])
>>> batch.schema
n_legs: int64
animals: string

select(self, columns)¶

Select columns of the RecordBatch.

Returns a new RecordBatch with the specified columns, and metadata preserved.

Parameters:

columnslist-like: The column names or integer indices to select.

Returns:

RecordBatch

Examples

>>> import pyarrow as pa
>>> n_legs = pa.array([2, 2, 4, 4, 5, 100])
>>> animals = pa.array(["Flamingo", "Parrot", "Dog", "Horse", "Brittle stars", "Centipede"])
>>> batch = pa.record_batch([n_legs, animals],
...                          names=["n_legs", "animals"])

Select columns my indices:

>>> batch.select([1])
pyarrow.RecordBatch
animals: string
----
animals: ["Flamingo","Parrot","Dog","Horse","Brittle stars","Centipede"]

Select columns by names:

>>> batch.select(["n_legs"])
pyarrow.RecordBatch
n_legs: int64
----
n_legs: [2,2,4,4,5,100]

serialize(self, memory_pool=None)¶

Write RecordBatch to Buffer as encapsulated IPC message, which does not include a Schema.

To reconstruct a RecordBatch from the encapsulated IPC message Buffer returned by this function, a Schema must be passed separately. See Examples.

Parameters:

memory_poolMemoryPool, default None: Uses default memory pool if not specified

Returns:

serializedBuffer

Examples

>>> import pyarrow as pa
>>> n_legs = pa.array([2, 2, 4, 4, 5, 100])
>>> animals = pa.array(["Flamingo", "Parrot", "Dog", "Horse", "Brittle stars", "Centipede"])
>>> batch = pa.RecordBatch.from_arrays([n_legs, animals],
...                                     names=["n_legs", "animals"])
>>> buf = batch.serialize()
>>> buf
<pyarrow.Buffer address=0x... size=... is_cpu=True is_mutable=True>

Reconstruct RecordBatch from IPC message Buffer and original Schema

>>> pa.ipc.read_record_batch(buf, batch.schema)
pyarrow.RecordBatch
n_legs: int64
animals: string
----
n_legs: [2,2,4,4,5,100]
animals: ["Flamingo","Parrot","Dog","Horse","Brittle stars","Centipede"]

slice(self, offset=0, length=None)¶

Compute zero-copy slice of this RecordBatch

Parameters:

offsetint, default 0: Offset from start of record batch to slice
lengthint, default None: Length of slice (default is until end of batch starting from offset)

Returns:

slicedRecordBatch

Examples

>>> import pyarrow as pa
>>> n_legs = pa.array([2, 2, 4, 4, 5, 100])
>>> animals = pa.array(["Flamingo", "Parrot", "Dog", "Horse", "Brittle stars", "Centipede"])
>>> batch = pa.RecordBatch.from_arrays([n_legs, animals],
...                                     names=["n_legs", "animals"])
>>> batch.to_pandas()
   n_legs        animals
0       2       Flamingo
1       2         Parrot
2       4            Dog
3       4          Horse
4       5  Brittle stars
5     100      Centipede
>>> batch.slice(offset=3).to_pandas()
   n_legs        animals
0       4          Horse
1       5  Brittle stars
2     100      Centipede
>>> batch.slice(length=2).to_pandas()
   n_legs   animals
0       2  Flamingo
1       2    Parrot
>>> batch.slice(offset=3, length=1).to_pandas()
   n_legs animals
0       4   Horse

sort_by(self, sorting, **kwargs)¶

Sort the Table or RecordBatch by one or multiple columns.

Parameters:

sortingstr or list[tuple(name, order)]: Name of the column to use to sort (ascending), or a list of multiple sorting conditions where each entry is a tuple with column name and sorting order (“ascending” or “descending”)
**kwargsdict, optional: Additional sorting options. As allowed by SortOptions

Returns:

Table or RecordBatch: A new tabular object sorted according to the sort keys.

Examples

Table (works similarly for RecordBatch)

>>> import pandas as pd
>>> import pyarrow as pa
>>> df = pd.DataFrame({'year': [2020, 2022, 2021, 2022, 2019, 2021],
...                    'n_legs': [2, 2, 4, 4, 5, 100],
...                    'animal': ["Flamingo", "Parrot", "Dog", "Horse",
...                    "Brittle stars", "Centipede"]})
>>> table = pa.Table.from_pandas(df)
>>> table.sort_by('animal')
pyarrow.Table
year: int64
n_legs: int64
animal: string
----
year: [[2019,2021,2021,2020,2022,2022]]
n_legs: [[5,100,4,2,4,2]]
animal: [["Brittle stars","Centipede","Dog","Flamingo","Horse","Parrot"]]

take(self, indices)¶

Select rows from a Table or RecordBatch.

See pyarrow.compute.take() for full usage.

Parameters:

indicesArray or array-like: The indices in the tabular object whose rows will be returned.

Returns:

Table or RecordBatch: A tabular object with the same schema, containing the taken rows.

Examples

Table (works similarly for RecordBatch)

>>> import pyarrow as pa
>>> import pandas as pd
>>> df = pd.DataFrame({'year': [2020, 2022, 2019, 2021],
...                    'n_legs': [2, 4, 5, 100],
...                    'animals': ["Flamingo", "Horse", "Brittle stars", "Centipede"]})
>>> table = pa.Table.from_pandas(df)
>>> table.take([1,3])
pyarrow.Table
year: int64
n_legs: int64
animals: string
----
year: [[2022,2021]]
n_legs: [[4,100]]
animals: [["Horse","Centipede"]]

to_pandas(self, memory_pool=None, categories=None, bool strings_to_categorical=False, bool zero_copy_only=False, bool integer_object_nulls=False, bool date_as_object=True, bool timestamp_as_object=False, bool use_threads=True, bool deduplicate_objects=True, bool ignore_metadata=False, bool safe=True, bool split_blocks=False, bool self_destruct=False, unicode maps_as_pydicts=None, types_mapper=None, bool coerce_temporal_nanoseconds=False)¶

Convert to a pandas-compatible NumPy array or DataFrame, as appropriate

Parameters:

memory_poolMemoryPool, default None

Arrow MemoryPool to use for allocations. Uses the default memory pool if not passed.

categorieslist, default empty

List of fields that should be returned as pandas.Categorical. Only applies to table-like data structures.

strings_to_categoricalbool, default False

Encode string (UTF8) and binary types to pandas.Categorical.

zero_copy_onlybool, default False

Raise an ArrowException if this function call would require copying the underlying data.

integer_object_nullsbool, default False

Cast integers with nulls to objects

date_as_objectbool, default True

Cast dates to objects. If False, convert to datetime64 dtype with the equivalent time unit (if supported). Note: in pandas version < 2.0, only datetime64[ns] conversion is supported.

timestamp_as_objectbool, default False

Cast non-nanosecond timestamps (np.datetime64) to objects. This is useful in pandas version 1.x if you have timestamps that don’t fit in the normal date range of nanosecond timestamps (1678 CE-2262 CE). Non-nanosecond timestamps are supported in pandas version 2.0. If False, all timestamps are converted to datetime64 dtype.

use_threadsbool, default True

Whether to parallelize the conversion using multiple threads.

deduplicate_objectsbool, default True

Do not create multiple copies Python objects when created, to save on memory use. Conversion will be slower.

ignore_metadatabool, default False

If True, do not use the ‘pandas’ metadata to reconstruct the DataFrame index, if present

safebool, default True

For certain data types, a cast is needed in order to store the data in a pandas DataFrame or Series (e.g. timestamps are always stored as nanoseconds in pandas). This option controls whether it is a safe cast or not.

split_blocksbool, default False

If True, generate one internal “block” for each column when creating a pandas.DataFrame from a RecordBatch or Table. While this can temporarily reduce memory note that various pandas operations can trigger “consolidation” which may balloon memory use.

self_destructbool, default False

EXPERIMENTAL: If True, attempt to deallocate the originating Arrow memory while converting the Arrow object to pandas. If you use the object after calling to_pandas with this option it will crash your program.

Note that you may not see always memory usage improvements. For example, if multiple columns share an underlying allocation, memory can’t be freed until all columns are converted.

maps_as_pydictsstr, optional, default None

Valid values are None, ‘lossy’, or ‘strict’. The default behavior (None), is to convert Arrow Map arrays to Python association lists (list-of-tuples) in the same order as the Arrow Map, as in [(key1, value1), (key2, value2), …].

If ‘lossy’ or ‘strict’, convert Arrow Map arrays to native Python dicts. This can change the ordering of (key, value) pairs, and will deduplicate multiple keys, resulting in a possible loss of data.

If ‘lossy’, this key deduplication results in a warning printed when detected. If ‘strict’, this instead results in an exception being raised when detected.

types_mapperfunction, default None

A function mapping a pyarrow DataType to a pandas ExtensionDtype. This can be used to override the default pandas type for conversion of built-in pyarrow types or in absence of pandas_metadata in the Table schema. The function receives a pyarrow DataType and is expected to return a pandas ExtensionDtype or None if the default conversion should be used for that type. If you have a dictionary mapping, you can pass dict.get as function.

coerce_temporal_nanosecondsbool, default False

Only applicable to pandas version >= 2.0. A legacy option to coerce date32, date64, duration, and timestamp time units to nanoseconds when converting to pandas. This is the default behavior in pandas version 1.x. Set this option to True if you’d like to use this coercion when using pandas version >= 2.0 for backwards compatibility (not recommended otherwise).

Returns:

pandas.Series or pandas.DataFrame depending on type of object

Examples

>>> import pyarrow as pa
>>> import pandas as pd

Convert a Table to pandas DataFrame:

>>> table = pa.table([
...    pa.array([2, 4, 5, 100]),
...    pa.array(["Flamingo", "Horse", "Brittle stars", "Centipede"])
...    ], names=['n_legs', 'animals'])
>>> table.to_pandas()
   n_legs        animals
0       2       Flamingo
1       4          Horse
2       5  Brittle stars
3     100      Centipede
>>> isinstance(table.to_pandas(), pd.DataFrame)
True

Convert a RecordBatch to pandas DataFrame:

>>> import pyarrow as pa
>>> n_legs = pa.array([2, 4, 5, 100])
>>> animals = pa.array(["Flamingo", "Horse", "Brittle stars", "Centipede"])
>>> batch = pa.record_batch([n_legs, animals],
...                         names=["n_legs", "animals"])
>>> batch
pyarrow.RecordBatch
n_legs: int64
animals: string
----
n_legs: [2,4,5,100]
animals: ["Flamingo","Horse","Brittle stars","Centipede"]
>>> batch.to_pandas()
   n_legs        animals
0       2       Flamingo
1       4          Horse
2       5  Brittle stars
3     100      Centipede
>>> isinstance(batch.to_pandas(), pd.DataFrame)
True

Convert a Chunked Array to pandas Series:

>>> import pyarrow as pa
>>> n_legs = pa.chunked_array([[2, 2, 4], [4, 5, 100]])
>>> n_legs.to_pandas()
0      2
1      2
2      4
3      4
4      5
5    100
dtype: int64
>>> isinstance(n_legs.to_pandas(), pd.Series)
True

to_pydict(self)¶

Convert the Table or RecordBatch to a dict or OrderedDict.

Returns:

dict

Examples

Table (works similarly for RecordBatch)

>>> import pyarrow as pa
>>> n_legs = pa.array([2, 2, 4, 4, 5, 100])
>>> animals = pa.array(["Flamingo", "Parrot", "Dog", "Horse", "Brittle stars", "Centipede"])
>>> table = pa.Table.from_arrays([n_legs, animals], names=["n_legs", "animals"])
>>> table.to_pydict()
{'n_legs': [2, 2, 4, 4, 5, 100], 'animals': ['Flamingo', 'Parrot', ..., 'Centipede']}

to_pylist(self)¶

Convert the Table or RecordBatch to a list of rows / dictionaries.

Returns:

list

Examples

Table (works similarly for RecordBatch)

>>> import pyarrow as pa
>>> data = [[2, 4, 5, 100],
...         ["Flamingo", "Horse", "Brittle stars", "Centipede"]]
>>> table = pa.table(data, names=["n_legs", "animals"])
>>> table.to_pylist()
[{'n_legs': 2, 'animals': 'Flamingo'}, {'n_legs': 4, 'animals': 'Horse'}, ...

to_string(self, *, show_metadata=False, preview_cols=0)¶

Return human-readable string representation of Table or RecordBatch.

Parameters:

show_metadatabool, default False: Display Field-level and Schema-level KeyValueMetadata.
preview_colsint, default 0: Display values of the columns for the first N columns.

Returns:

str

to_struct_array(self)¶: Convert to a struct array.

validate(self, *, full=False)¶

Perform validation checks. An exception is raised if validation fails.

By default only cheap validation checks are run. Pass full=True for thorough validation checks (potentially O(n)).

Parameters:

fullbool, default False: If True, run expensive checks, otherwise cheap checks only.

Raises:

ArrowInvalid

pyarrow.ChunkedArray

pyarrow.Table