pyarrow.RecordBatch

class pyarrow.RecordBatch

Bases: pyarrow.lib._PandasConvertible

Batch of rows of columns of equal length

Warning

Do not call this class’s constructor directly, use one of the RecordBatch.from_* functions instead.

__init__(*args, **kwargs)

Initialize self. See help(type(self)) for accurate signature.

Methods

__init__(*args, **kwargs)

Initialize self.

column(self, i)

Select single column from record batch

equals(self, other, bool check_metadata=False)

Check if contents of two record batches are equal.

field(self, i)

Select a schema field by its column name or numeric index

filter(self, Array mask[, …])

Select record from a record batch.

from_arrays(list arrays[, names, schema, …])

Construct a RecordBatch from multiple pyarrow.Arrays

from_pandas(type cls, df, Schema schema=None)

Convert pandas.DataFrame to an Arrow RecordBatch

from_struct_array(StructArray struct_array)

Construct a RecordBatch from a StructArray.

replace_schema_metadata(self[, metadata])

Create shallow copy of record batch by replacing schema key-value metadata with the indicated new metadata (which may be None, which deletes any existing metadata

serialize(self[, memory_pool])

Write RecordBatch to Buffer as encapsulated IPC message.

slice(self[, offset, length])

Compute zero-copy slice of this RecordBatch

take(self, indices)

Select records from an RecordBatch.

to_pandas(self[, memory_pool, categories, …])

Convert to a pandas-compatible NumPy array or DataFrame, as appropriate

to_pydict(self)

Convert the RecordBatch to a dict or OrderedDict.

to_string(self[, show_metadata])

validate(self, *[, full])

Perform validation checks.

Attributes

columns

List of all columns in numerical order

nbytes

Total number of bytes consumed by the elements of the record batch.

num_columns

Number of columns

num_rows

Number of rows

schema

Schema of the RecordBatch and its columns
column(self, i)

Select single column from record batch

Parameters

i (int or string) – The index or name of the column to retrieve.

Returns

column (pyarrow.Array)

columns

List of all columns in numerical order

Returns

list of pa.Array

equals(self, other, bool check_metadata=False)

Check if contents of two record batches are equal.

Parameters

  • other (pyarrow.RecordBatch) – RecordBatch to compare against.

  • check_metadata (bool, default False) – Whether schema metadata equality should be checked as well.

Returns

are_equal (bool)

field(self, i)

Select a schema field by its column name or numeric index

Parameters

i (int or string) – The index or name of the field to retrieve

Returns

pyarrow.Field

filter(self, Array mask, null_selection_behavior=u'drop')

Select record from a record batch. See pyarrow.compute.filter for full usage.

static from_arrays(list arrays, names=None, schema=None, metadata=None)

Construct a RecordBatch from multiple pyarrow.Arrays

Parameters

  • arrays (list of pyarrow.Array) – One for each field in RecordBatch

  • names (list of str, optional) – Names for the batch fields. If not passed, schema must be passed

  • schema (Schema, default None) – Schema for the created batch. If not passed, names must be passed

  • metadata (dict or Mapping, default None) – Optional metadata for the schema (if inferred).

Returns

pyarrow.RecordBatch

from_pandas(type cls, df, Schema schema=None, preserve_index=None, nthreads=None, columns=None)

Convert pandas.DataFrame to an Arrow RecordBatch

Parameters

  • df (pandas.DataFrame) –

  • schema (pyarrow.Schema, optional) – The expected schema of the RecordBatch. This can be used to indicate the type of columns if we cannot infer it automatically. If passed, the output will have exactly this schema. Columns specified in the schema that are not found in the DataFrame columns or its index will raise an error. Additional columns or index levels in the DataFrame which are not specified in the schema will be ignored.

  • preserve_index (bool, optional) – Whether to store the index as an additional column in the resulting RecordBatch. The default of None will store the index as a column, except for RangeIndex which is stored as metadata only. Use preserve_index=True to force it to be stored as a column.

  • nthreads (int, default None (may use up to system CPU count threads)) – If greater than 1, convert columns to Arrow in parallel using indicated number of threads

  • columns (list, optional) – List of column to be converted. If None, use all columns.

Returns

pyarrow.RecordBatch

static from_struct_array(StructArray struct_array)

Construct a RecordBatch from a StructArray.

Each field in the StructArray will become a column in the resulting RecordBatch.

Parameters

struct_array (StructArray) – Array to construct the record batch from.

Returns

pyarrow.RecordBatch

nbytes

Total number of bytes consumed by the elements of the record batch.

num_columns

Number of columns

Returns

int

num_rows

Number of rows

Due to the definition of a RecordBatch, all columns have the same number of rows.

Returns

int

replace_schema_metadata(self, metadata=None)

Create shallow copy of record batch by replacing schema key-value metadata with the indicated new metadata (which may be None, which deletes any existing metadata

Parameters

metadata (dict, default None) –

Returns

shallow_copy (RecordBatch)

schema

Schema of the RecordBatch and its columns

Returns

pyarrow.Schema

serialize(self, memory_pool=None)

Write RecordBatch to Buffer as encapsulated IPC message.

Parameters

memory_pool (MemoryPool, default None) – Uses default memory pool if not specified

Returns

serialized (Buffer)

slice(self, offset=0, length=None)

Compute zero-copy slice of this RecordBatch

Parameters

  • offset (int, default 0) – Offset from start of record batch to slice

  • length (int, default None) – Length of slice (default is until end of batch starting from offset)

Returns

sliced (RecordBatch)

take(self, indices)

Select records from an RecordBatch. See pyarrow.compute.take for full usage.

to_pandas(self, memory_pool=None, categories=None, bool strings_to_categorical=False, bool zero_copy_only=False, bool integer_object_nulls=False, bool date_as_object=True, bool timestamp_as_object=False, bool use_threads=True, bool deduplicate_objects=True, bool ignore_metadata=False, bool safe=True, bool split_blocks=False, bool self_destruct=False, types_mapper=None)

Convert to a pandas-compatible NumPy array or DataFrame, as appropriate

Parameters

  • memory_pool (MemoryPool, default None) – Arrow MemoryPool to use for allocations. Uses the default memory pool is not passed.

  • strings_to_categorical (bool, default False) – Encode string (UTF8) and binary types to pandas.Categorical.

  • categories (list, default empty) – List of fields that should be returned as pandas.Categorical. Only applies to table-like data structures.

  • zero_copy_only (bool, default False) – Raise an ArrowException if this function call would require copying the underlying data.

  • integer_object_nulls (bool, default False) – Cast integers with nulls to objects

  • date_as_object (bool, default True) – Cast dates to objects. If False, convert to datetime64[ns] dtype.

  • timestamp_as_object (bool, default False) – Cast non-nanosecond timestamps (np.datetime64) to objects. This is useful if you have timestamps that don’t fit in the normal date range of nanosecond timestamps (1678 CE-2262 CE). If False, all timestamps are converted to datetime64[ns] dtype.

  • use_threads (bool, default True) – Whether to parallelize the conversion using multiple threads.

  • deduplicate_objects (bool, default False) – Do not create multiple copies Python objects when created, to save on memory use. Conversion will be slower.

  • ignore_metadata (bool, default False) – If True, do not use the ‘pandas’ metadata to reconstruct the DataFrame index, if present

  • safe (bool, default True) – For certain data types, a cast is needed in order to store the data in a pandas DataFrame or Series (e.g. timestamps are always stored as nanoseconds in pandas). This option controls whether it is a safe cast or not.

  • split_blocks (bool, default False) – If True, generate one internal “block” for each column when creating a pandas.DataFrame from a RecordBatch or Table. While this can temporarily reduce memory note that various pandas operations can trigger “consolidation” which may balloon memory use.

  • self_destruct (bool, default False) –

    EXPERIMENTAL: If True, attempt to deallocate the originating Arrow memory while converting the Arrow object to pandas. If you use the object after calling to_pandas with this option it will crash your program.

    Note that you may not see always memory usage improvements. For example, if multiple columns share an underlying allocation, memory can’t be freed until all columns are converted.

  • types_mapper (function, default None) – A function mapping a pyarrow DataType to a pandas ExtensionDtype. This can be used to override the default pandas type for conversion of built-in pyarrow types or in absence of pandas_metadata in the Table schema. The function receives a pyarrow DataType and is expected to return a pandas ExtensionDtype or None if the default conversion should be used for that type. If you have a dictionary mapping, you can pass dict.get as function.

Returns

pandas.Series or pandas.DataFrame depending on type of object

to_pydict(self)

Convert the RecordBatch to a dict or OrderedDict.

Returns

dict

to_string(self, show_metadata=False)
validate(self, *, full=False)

Perform validation checks. An exception is raised if validation fails.

By default only cheap validation checks are run. Pass full=True for thorough validation checks (potentially O(n)).

Parameters

full (bool, default False) – If True, run expensive checks, otherwise cheap checks only.

Raises

ArrowInvalid –

pyarrow.ChunkedArray pyarrow.Table