pyarrow.RecordBatch¶
- class pyarrow.RecordBatch¶
Bases:
_PandasConvertible
Batch of rows of columns of equal length
Warning
Do not call this class’s constructor directly, use one of the
RecordBatch.from_*
functions instead.Examples
>>> import pyarrow as pa >>> n_legs = pa.array([2, 2, 4, 4, 5, 100]) >>> animals = pa.array(["Flamingo", "Parrot", "Dog", "Horse", "Brittle stars", "Centipede"]) >>> names = ["n_legs", "animals"]
Constructing a RecordBatch from arrays:
>>> pa.RecordBatch.from_arrays([n_legs, animals], names=names) pyarrow.RecordBatch n_legs: int64 animals: string >>> pa.RecordBatch.from_arrays([n_legs, animals], names=names).to_pandas() n_legs animals 0 2 Flamingo 1 2 Parrot 2 4 Dog 3 4 Horse 4 5 Brittle stars 5 100 Centipede
Constructing a RecordBatch from pandas DataFrame:
>>> import pandas as pd >>> df = pd.DataFrame({'year': [2020, 2022, 2021, 2022], ... 'month': [3, 5, 7, 9], ... 'day': [1, 5, 9, 13], ... 'n_legs': [2, 4, 5, 100], ... 'animals': ["Flamingo", "Horse", "Brittle stars", "Centipede"]}) >>> pa.RecordBatch.from_pandas(df) pyarrow.RecordBatch year: int64 month: int64 day: int64 n_legs: int64 animals: string >>> pa.RecordBatch.from_pandas(df).to_pandas() year month day n_legs animals 0 2020 3 1 2 Flamingo 1 2022 5 5 4 Horse 2 2021 7 9 5 Brittle stars 3 2022 9 13 100 Centipede
Constructing a RecordBatch from pylist:
>>> pylist = [{'n_legs': 2, 'animals': 'Flamingo'}, ... {'n_legs': 4, 'animals': 'Dog'}] >>> pa.RecordBatch.from_pylist(pylist).to_pandas() n_legs animals 0 2 Flamingo 1 4 Dog
You can also construct a RecordBatch using
pyarrow.record_batch()
:>>> pa.record_batch([n_legs, animals], names=names).to_pandas() n_legs animals 0 2 Flamingo 1 2 Parrot 2 4 Dog 3 4 Horse 4 5 Brittle stars 5 100 Centipede
>>> pa.record_batch(df) pyarrow.RecordBatch year: int64 month: int64 day: int64 n_legs: int64 animals: string
- __init__(*args, **kwargs)¶
Methods
__init__
(*args, **kwargs)column
(self, i)Select single column from record batch
drop_null
(self)Remove missing values from a RecordBatch.
equals
(self, other, bool check_metadata=False)Check if contents of two record batches are equal.
field
(self, i)Select a schema field by its column name or numeric index
filter
(self, mask[, null_selection_behavior])Select rows from the record batch.
from_arrays
(list arrays[, names, schema, ...])Construct a RecordBatch from multiple pyarrow.Arrays
from_pandas
(type cls, df, Schema schema=None)Convert pandas.DataFrame to an Arrow RecordBatch
from_pydict
(mapping[, schema, metadata])Construct a RecordBatch from Arrow arrays or columns.
from_pylist
(mapping[, schema, metadata])Construct a RecordBatch from list of rows / dictionaries.
from_struct_array
(StructArray struct_array)Construct a RecordBatch from a StructArray.
get_total_buffer_size
(self)The sum of bytes in each buffer referenced by the record batch
replace_schema_metadata
(self[, metadata])Create shallow copy of record batch by replacing schema key-value metadata with the indicated new metadata (which may be None, which deletes any existing metadata
serialize
(self[, memory_pool])Write RecordBatch to Buffer as encapsulated IPC message.
slice
(self[, offset, length])Compute zero-copy slice of this RecordBatch
take
(self, indices)Select rows from the record batch.
to_pandas
(self[, memory_pool, categories, ...])Convert to a pandas-compatible NumPy array or DataFrame, as appropriate
to_pydict
(self)Convert the RecordBatch to a dict or OrderedDict.
to_pylist
(self)Convert the RecordBatch to a list of rows / dictionaries.
to_string
(self[, show_metadata])validate
(self, *[, full])Perform validation checks.
Attributes
List of all columns in numerical order
Total number of bytes consumed by the elements of the record batch.
Number of columns
Number of rows
Schema of the RecordBatch and its columns
- column(self, i)¶
Select single column from record batch
- Parameters:
- Returns:
- column
pyarrow.Array
- column
Examples
>>> import pyarrow as pa >>> n_legs = pa.array([2, 2, 4, 4, 5, 100]) >>> animals = pa.array(["Flamingo", "Parrot", "Dog", "Horse", "Brittle stars", "Centipede"]) >>> batch = pa.RecordBatch.from_arrays([n_legs, animals], ... names=["n_legs", "animals"]) >>> batch.column(1) <pyarrow.lib.StringArray object at ...> [ "Flamingo", "Parrot", "Dog", "Horse", "Brittle stars", "Centipede" ]
- columns¶
List of all columns in numerical order
- Returns:
Examples
>>> import pyarrow as pa >>> n_legs = pa.array([2, 2, 4, 4, 5, 100]) >>> animals = pa.array(["Flamingo", "Parrot", "Dog", "Horse", "Brittle stars", "Centipede"]) >>> batch = pa.RecordBatch.from_arrays([n_legs, animals], ... names=["n_legs", "animals"]) >>> batch.columns [<pyarrow.lib.Int64Array object at ...> [ 2, 2, 4, 4, 5, 100 ], <pyarrow.lib.StringArray object at ...> [ "Flamingo", "Parrot", "Dog", "Horse", "Brittle stars", "Centipede" ]]
- drop_null(self)¶
Remove missing values from a RecordBatch. See
pyarrow.compute.drop_null()
for full usage.Examples
>>> import pyarrow as pa >>> n_legs = pa.array([2, 2, 4, 4, 5, 100]) >>> animals = pa.array(["Flamingo", "Parrot", "Dog", "Horse", None, "Centipede"]) >>> batch = pa.RecordBatch.from_arrays([n_legs, animals], ... names=["n_legs", "animals"]) >>> batch.to_pandas() n_legs animals 0 2 Flamingo 1 2 Parrot 2 4 Dog 3 4 Horse 4 5 None 5 100 Centipede >>> batch.drop_null().to_pandas() n_legs animals 0 2 Flamingo 1 2 Parrot 2 4 Dog 3 4 Horse 4 100 Centipede
- equals(self, other, bool check_metadata=False)¶
Check if contents of two record batches are equal.
- Parameters:
- other
pyarrow.RecordBatch
RecordBatch to compare against.
- check_metadatabool, default
False
Whether schema metadata equality should be checked as well.
- other
- Returns:
- are_equalbool
Examples
>>> import pyarrow as pa >>> n_legs = pa.array([2, 2, 4, 4, 5, 100]) >>> animals = pa.array(["Flamingo", "Parrot", "Dog", "Horse", "Brittle stars", "Centipede"]) >>> batch = pa.RecordBatch.from_arrays([n_legs, animals], ... names=["n_legs", "animals"]) >>> batch_0 = pa.record_batch([]) >>> batch_1 = pa.RecordBatch.from_arrays([n_legs, animals], ... names=["n_legs", "animals"], ... metadata={"n_legs": "Number of legs per animal"}) >>> batch.equals(batch) True >>> batch.equals(batch_0) False >>> batch.equals(batch_1) True >>> batch.equals(batch_1, check_metadata=True) False
- field(self, i)¶
Select a schema field by its column name or numeric index
- Parameters:
- Returns:
Examples
>>> import pyarrow as pa >>> n_legs = pa.array([2, 2, 4, 4, 5, 100]) >>> animals = pa.array(["Flamingo", "Parrot", "Dog", "Horse", "Brittle stars", "Centipede"]) >>> batch = pa.RecordBatch.from_arrays([n_legs, animals], ... names=["n_legs", "animals"]) >>> batch.field(0) pyarrow.Field<n_legs: int64> >>> batch.field(1) pyarrow.Field<animals: string>
- filter(self, mask, null_selection_behavior='drop')¶
Select rows from the record batch.
See
pyarrow.compute.filter()
for full usage.- Parameters:
- mask
Array
orarray-like
The boolean mask to filter the record batch with.
- null_selection_behavior
How nulls in the mask should be handled.
- mask
- Returns:
- filtered
RecordBatch
A record batch of the same schema, with only the rows selected by the boolean mask.
- filtered
Examples
>>> import pyarrow as pa >>> n_legs = pa.array([2, 2, 4, 4, 5, 100]) >>> animals = pa.array(["Flamingo", "Parrot", "Dog", "Horse", "Brittle stars", "Centipede"]) >>> batch = pa.RecordBatch.from_arrays([n_legs, animals], ... names=["n_legs", "animals"]) >>> batch.to_pandas() n_legs animals 0 2 Flamingo 1 2 Parrot 2 4 Dog 3 4 Horse 4 5 Brittle stars 5 100 Centipede
Define a mask and select rows:
>>> mask=[True, True, False, True, False, None] >>> batch.filter(mask).to_pandas() n_legs animals 0 2 Flamingo 1 2 Parrot 2 4 Horse >>> batch.filter(mask, null_selection_behavior='emit_null').to_pandas() n_legs animals 0 2.0 Flamingo 1 2.0 Parrot 2 4.0 Horse 3 NaN None
- static from_arrays(list arrays, names=None, schema=None, metadata=None)¶
Construct a RecordBatch from multiple pyarrow.Arrays
- Parameters:
- arrays
list
ofpyarrow.Array
One for each field in RecordBatch
- names
list
ofstr
, optional Names for the batch fields. If not passed, schema must be passed
- schema
Schema
, defaultNone
Schema for the created batch. If not passed, names must be passed
- metadata
dict
or Mapping, defaultNone
Optional metadata for the schema (if inferred).
- arrays
- Returns:
Examples
>>> import pyarrow as pa >>> n_legs = pa.array([2, 2, 4, 4, 5, 100]) >>> animals = pa.array(["Flamingo", "Parrot", "Dog", "Horse", "Brittle stars", "Centipede"]) >>> names = ["n_legs", "animals"]
Construct a RecordBartch from pyarrow Arrays using names:
>>> pa.RecordBatch.from_arrays([n_legs, animals], names=names) pyarrow.RecordBatch n_legs: int64 animals: string >>> pa.RecordBatch.from_arrays([n_legs, animals], names=names).to_pandas() n_legs animals 0 2 Flamingo 1 2 Parrot 2 4 Dog 3 4 Horse 4 5 Brittle stars 5 100 Centipede
Construct a RecordBartch from pyarrow Arrays using schema:
>>> my_schema = pa.schema([ ... pa.field('n_legs', pa.int64()), ... pa.field('animals', pa.string())], ... metadata={"n_legs": "Number of legs per animal"}) >>> pa.RecordBatch.from_arrays([n_legs, animals], schema=my_schema).to_pandas() n_legs animals 0 2 Flamingo 1 2 Parrot 2 4 Dog 3 4 Horse 4 5 Brittle stars 5 100 Centipede >>> pa.RecordBatch.from_arrays([n_legs, animals], schema=my_schema).schema n_legs: int64 animals: string -- schema metadata -- n_legs: 'Number of legs per animal'
- from_pandas(type cls, df, Schema schema=None, preserve_index=None, nthreads=None, columns=None)¶
Convert pandas.DataFrame to an Arrow RecordBatch
- Parameters:
- df
pandas.DataFrame
- schema
pyarrow.Schema
, optional The expected schema of the RecordBatch. This can be used to indicate the type of columns if we cannot infer it automatically. If passed, the output will have exactly this schema. Columns specified in the schema that are not found in the DataFrame columns or its index will raise an error. Additional columns or index levels in the DataFrame which are not specified in the schema will be ignored.
- preserve_indexbool, optional
Whether to store the index as an additional column in the resulting
RecordBatch
. The default of None will store the index as a column, except for RangeIndex which is stored as metadata only. Usepreserve_index=True
to force it to be stored as a column.- nthreads
int
, defaultNone
If greater than 1, convert columns to Arrow in parallel using indicated number of threads. By default, this follows
pyarrow.cpu_count()
(may use up to system CPU count threads).- columns
list
, optional List of column to be converted. If None, use all columns.
- df
- Returns:
Examples
>>> import pandas as pd >>> df = pd.DataFrame({'year': [2020, 2022, 2021, 2022], ... 'month': [3, 5, 7, 9], ... 'day': [1, 5, 9, 13], ... 'n_legs': [2, 4, 5, 100], ... 'animals': ["Flamingo", "Horse", "Brittle stars", "Centipede"]})
Convert pandas DataFrame to RecordBatch:
>>> import pyarrow as pa >>> pa.RecordBatch.from_pandas(df) pyarrow.RecordBatch year: int64 month: int64 day: int64 n_legs: int64 animals: string
Convert pandas DataFrame to RecordBatch using schema:
>>> my_schema = pa.schema([ ... pa.field('n_legs', pa.int64()), ... pa.field('animals', pa.string())], ... metadata={"n_legs": "Number of legs per animal"}) >>> pa.RecordBatch.from_pandas(df, schema=my_schema) pyarrow.RecordBatch n_legs: int64 animals: string
Convert pandas DataFrame to RecordBatch specifying columns:
>>> pa.RecordBatch.from_pandas(df, columns=["n_legs"]) pyarrow.RecordBatch n_legs: int64
- static from_pydict(mapping, schema=None, metadata=None)¶
Construct a RecordBatch from Arrow arrays or columns.
- Parameters:
- Returns:
Examples
>>> import pyarrow as pa >>> n_legs = [2, 2, 4, 4, 5, 100] >>> animals = ["Flamingo", "Parrot", "Dog", "Horse", "Brittle stars", "Centipede"] >>> pydict = {'n_legs': n_legs, 'animals': animals}
Construct a RecordBatch from arrays:
>>> pa.RecordBatch.from_pydict(pydict) pyarrow.RecordBatch n_legs: int64 animals: string >>> pa.RecordBatch.from_pydict(pydict).to_pandas() n_legs animals 0 2 Flamingo 1 2 Parrot 2 4 Dog 3 4 Horse 4 5 Brittle stars 5 100 Centipede
Construct a RecordBatch with schema:
>>> my_schema = pa.schema([ ... pa.field('n_legs', pa.int64()), ... pa.field('animals', pa.string())], ... metadata={"n_legs": "Number of legs per animal"}) >>> pa.RecordBatch.from_pydict(pydict, schema=my_schema).schema n_legs: int64 animals: string -- schema metadata -- n_legs: 'Number of legs per animal'
- static from_pylist(mapping, schema=None, metadata=None)¶
Construct a RecordBatch from list of rows / dictionaries.
- Parameters:
- Returns:
Examples
>>> import pyarrow as pa >>> pylist = [{'n_legs': 2, 'animals': 'Flamingo'}, ... {'n_legs': 4, 'animals': 'Dog'}] >>> pa.RecordBatch.from_pylist(pylist) pyarrow.RecordBatch n_legs: int64 animals: string >>> pa.RecordBatch.from_pylist(pylist).to_pandas() n_legs animals 0 2 Flamingo 1 4 Dog
Construct a RecordBatch with metadata:
>>> my_metadata={"n_legs": "Number of legs per animal"} >>> pa.RecordBatch.from_pylist(pylist, metadata=my_metadata).schema n_legs: int64 animals: string -- schema metadata -- n_legs: 'Number of legs per animal'
- static from_struct_array(StructArray struct_array)¶
Construct a RecordBatch from a StructArray.
Each field in the StructArray will become a column in the resulting
RecordBatch
.- Parameters:
- struct_array
StructArray
Array to construct the record batch from.
- struct_array
- Returns:
Examples
>>> import pyarrow as pa >>> struct = pa.array([{'n_legs': 2, 'animals': 'Parrot'}, ... {'year': 2022, 'n_legs': 4}]) >>> pa.RecordBatch.from_struct_array(struct).to_pandas() animals n_legs year 0 Parrot 2 NaN 1 None 4 2022.0
- get_total_buffer_size(self)¶
The sum of bytes in each buffer referenced by the record batch
An array may only reference a portion of a buffer. This method will overestimate in this case and return the byte size of the entire buffer.
If a buffer is referenced multiple times then it will only be counted once.
Examples
>>> import pyarrow as pa >>> n_legs = pa.array([2, 2, 4, 4, 5, 100]) >>> animals = pa.array(["Flamingo", "Parrot", "Dog", "Horse", "Brittle stars", "Centipede"]) >>> batch = pa.RecordBatch.from_arrays([n_legs, animals], ... names=["n_legs", "animals"]) >>> batch.get_total_buffer_size() 120
- nbytes¶
Total number of bytes consumed by the elements of the record batch.
In other words, the sum of bytes from all buffer ranges referenced.
Unlike get_total_buffer_size this method will account for array offsets.
If buffers are shared between arrays then the shared portion will only be counted multiple times.
The dictionary of dictionary arrays will always be counted in their entirety even if the array only references a portion of the dictionary.
Examples
>>> import pyarrow as pa >>> n_legs = pa.array([2, 2, 4, 4, 5, 100]) >>> animals = pa.array(["Flamingo", "Parrot", "Dog", "Horse", "Brittle stars", "Centipede"]) >>> batch = pa.RecordBatch.from_arrays([n_legs, animals], ... names=["n_legs", "animals"]) >>> batch.nbytes 116
- num_columns¶
Number of columns
- Returns:
Examples
>>> import pyarrow as pa >>> n_legs = pa.array([2, 2, 4, 4, 5, 100]) >>> animals = pa.array(["Flamingo", "Parrot", "Dog", "Horse", "Brittle stars", "Centipede"]) >>> batch = pa.RecordBatch.from_arrays([n_legs, animals], ... names=["n_legs", "animals"]) >>> batch.num_columns 2
- num_rows¶
Number of rows
Due to the definition of a RecordBatch, all columns have the same number of rows.
- Returns:
Examples
>>> import pyarrow as pa >>> n_legs = pa.array([2, 2, 4, 4, 5, 100]) >>> animals = pa.array(["Flamingo", "Parrot", "Dog", "Horse", "Brittle stars", "Centipede"]) >>> batch = pa.RecordBatch.from_arrays([n_legs, animals], ... names=["n_legs", "animals"]) >>> batch.num_rows 6
- replace_schema_metadata(self, metadata=None)¶
Create shallow copy of record batch by replacing schema key-value metadata with the indicated new metadata (which may be None, which deletes any existing metadata
- Parameters:
- Returns:
- shallow_copy
RecordBatch
- shallow_copy
Examples
>>> import pyarrow as pa >>> n_legs = pa.array([2, 2, 4, 4, 5, 100])
Constructing a RecordBatch with schema and metadata:
>>> my_schema = pa.schema([ ... pa.field('n_legs', pa.int64())], ... metadata={"n_legs": "Number of legs per animal"}) >>> batch = pa.RecordBatch.from_arrays([n_legs], schema=my_schema) >>> batch.schema n_legs: int64 -- schema metadata -- n_legs: 'Number of legs per animal'
Shallow copy of a RecordBatch with deleted schema metadata:
>>> batch.replace_schema_metadata().schema n_legs: int64
- schema¶
Schema of the RecordBatch and its columns
- Returns:
Examples
>>> import pyarrow as pa >>> n_legs = pa.array([2, 2, 4, 4, 5, 100]) >>> animals = pa.array(["Flamingo", "Parrot", "Dog", "Horse", "Brittle stars", "Centipede"]) >>> batch = pa.RecordBatch.from_arrays([n_legs, animals], ... names=["n_legs", "animals"]) >>> batch.schema n_legs: int64 animals: string
- serialize(self, memory_pool=None)¶
Write RecordBatch to Buffer as encapsulated IPC message.
- Parameters:
- memory_pool
MemoryPool
, defaultNone
Uses default memory pool if not specified
- memory_pool
- Returns:
- serialized
Buffer
- serialized
Examples
>>> import pyarrow as pa >>> n_legs = pa.array([2, 2, 4, 4, 5, 100]) >>> animals = pa.array(["Flamingo", "Parrot", "Dog", "Horse", "Brittle stars", "Centipede"]) >>> batch = pa.RecordBatch.from_arrays([n_legs, animals], ... names=["n_legs", "animals"]) >>> batch.serialize() <pyarrow.lib.Buffer object at ...>
- slice(self, offset=0, length=None)¶
Compute zero-copy slice of this RecordBatch
- Parameters:
- Returns:
- sliced
RecordBatch
- sliced
Examples
>>> import pyarrow as pa >>> n_legs = pa.array([2, 2, 4, 4, 5, 100]) >>> animals = pa.array(["Flamingo", "Parrot", "Dog", "Horse", "Brittle stars", "Centipede"]) >>> batch = pa.RecordBatch.from_arrays([n_legs, animals], ... names=["n_legs", "animals"]) >>> batch.to_pandas() n_legs animals 0 2 Flamingo 1 2 Parrot 2 4 Dog 3 4 Horse 4 5 Brittle stars 5 100 Centipede >>> batch.slice(offset=3).to_pandas() n_legs animals 0 4 Horse 1 5 Brittle stars 2 100 Centipede >>> batch.slice(length=2).to_pandas() n_legs animals 0 2 Flamingo 1 2 Parrot >>> batch.slice(offset=3, length=1).to_pandas() n_legs animals 0 4 Horse
- take(self, indices)¶
Select rows from the record batch.
See
pyarrow.compute.take()
for full usage.- Parameters:
- indices
Array
orarray-like
The indices in the record batch whose rows will be returned.
- indices
- Returns:
- taken
RecordBatch
A record batch with the same schema, containing the taken rows.
- taken
Examples
>>> import pyarrow as pa >>> n_legs = pa.array([2, 2, 4, 4, 5, 100]) >>> animals = pa.array(["Flamingo", "Parrot", "Dog", "Horse", "Brittle stars", "Centipede"]) >>> batch = pa.RecordBatch.from_arrays([n_legs, animals], ... names=["n_legs", "animals"]) >>> batch.take([1,3,4]).to_pandas() n_legs animals 0 2 Parrot 1 4 Horse 2 5 Brittle stars
- to_pandas(self, memory_pool=None, categories=None, bool strings_to_categorical=False, bool zero_copy_only=False, bool integer_object_nulls=False, bool date_as_object=True, bool timestamp_as_object=False, bool use_threads=True, bool deduplicate_objects=True, bool ignore_metadata=False, bool safe=True, bool split_blocks=False, bool self_destruct=False, types_mapper=None)¶
Convert to a pandas-compatible NumPy array or DataFrame, as appropriate
- Parameters:
- memory_pool
MemoryPool
, defaultNone
Arrow MemoryPool to use for allocations. Uses the default memory pool is not passed.
- strings_to_categoricalbool, default
False
Encode string (UTF8) and binary types to pandas.Categorical.
- categories
list
, defaultempty
List of fields that should be returned as pandas.Categorical. Only applies to table-like data structures.
- zero_copy_onlybool, default
False
Raise an ArrowException if this function call would require copying the underlying data.
- integer_object_nullsbool, default
False
Cast integers with nulls to objects
- date_as_objectbool, default
True
Cast dates to objects. If False, convert to datetime64[ns] dtype.
- timestamp_as_objectbool, default
False
Cast non-nanosecond timestamps (np.datetime64) to objects. This is useful if you have timestamps that don’t fit in the normal date range of nanosecond timestamps (1678 CE-2262 CE). If False, all timestamps are converted to datetime64[ns] dtype.
- use_threadsbool, default
True
Whether to parallelize the conversion using multiple threads.
- deduplicate_objectsbool, default
False
Do not create multiple copies Python objects when created, to save on memory use. Conversion will be slower.
- ignore_metadatabool, default
False
If True, do not use the ‘pandas’ metadata to reconstruct the DataFrame index, if present
- safebool, default
True
For certain data types, a cast is needed in order to store the data in a pandas DataFrame or Series (e.g. timestamps are always stored as nanoseconds in pandas). This option controls whether it is a safe cast or not.
- split_blocksbool, default
False
If True, generate one internal “block” for each column when creating a pandas.DataFrame from a RecordBatch or Table. While this can temporarily reduce memory note that various pandas operations can trigger “consolidation” which may balloon memory use.
- self_destructbool, default
False
EXPERIMENTAL: If True, attempt to deallocate the originating Arrow memory while converting the Arrow object to pandas. If you use the object after calling to_pandas with this option it will crash your program.
Note that you may not see always memory usage improvements. For example, if multiple columns share an underlying allocation, memory can’t be freed until all columns are converted.
- types_mapperfunction, default
None
A function mapping a pyarrow DataType to a pandas ExtensionDtype. This can be used to override the default pandas type for conversion of built-in pyarrow types or in absence of pandas_metadata in the Table schema. The function receives a pyarrow DataType and is expected to return a pandas ExtensionDtype or
None
if the default conversion should be used for that type. If you have a dictionary mapping, you can passdict.get
as function.
- memory_pool
- Returns:
pandas.Series
orpandas.DataFrame
depending ontype
of object
Examples
>>> import pyarrow as pa >>> import pandas as pd
Convert a Table to pandas DataFrame:
>>> table = pa.table([ ... pa.array([2, 4, 5, 100]), ... pa.array(["Flamingo", "Horse", "Brittle stars", "Centipede"]) ... ], names=['n_legs', 'animals']) >>> table.to_pandas() n_legs animals 0 2 Flamingo 1 4 Horse 2 5 Brittle stars 3 100 Centipede >>> isinstance(table.to_pandas(), pd.DataFrame) True
Convert a RecordBatch to pandas DataFrame:
>>> import pyarrow as pa >>> n_legs = pa.array([2, 4, 5, 100]) >>> animals = pa.array(["Flamingo", "Horse", "Brittle stars", "Centipede"]) >>> batch = pa.record_batch([n_legs, animals], ... names=["n_legs", "animals"]) >>> batch pyarrow.RecordBatch n_legs: int64 animals: string >>> batch.to_pandas() n_legs animals 0 2 Flamingo 1 4 Horse 2 5 Brittle stars 3 100 Centipede >>> isinstance(batch.to_pandas(), pd.DataFrame) True
Convert a Chunked Array to pandas Series:
>>> import pyarrow as pa >>> n_legs = pa.chunked_array([[2, 2, 4], [4, 5, 100]]) >>> n_legs.to_pandas() 0 2 1 2 2 4 3 4 4 5 5 100 dtype: int64 >>> isinstance(n_legs.to_pandas(), pd.Series) True
- to_pydict(self)¶
Convert the RecordBatch to a dict or OrderedDict.
- Returns:
Examples
>>> import pyarrow as pa >>> n_legs = pa.array([2, 2, 4, 4, 5, 100]) >>> animals = pa.array(["Flamingo", "Parrot", "Dog", "Horse", "Brittle stars", "Centipede"]) >>> batch = pa.RecordBatch.from_arrays([n_legs, animals], ... names=["n_legs", "animals"]) >>> batch.to_pydict() {'n_legs': [2, 2, 4, 4, 5, 100], 'animals': ['Flamingo', 'Parrot', ..., 'Centipede']}
- to_pylist(self)¶
Convert the RecordBatch to a list of rows / dictionaries.
- Returns:
Examples
>>> import pyarrow as pa >>> n_legs = pa.array([2, 2, 4, 4, 5, 100]) >>> animals = pa.array(["Flamingo", "Parrot", "Dog", "Horse", "Brittle stars", "Centipede"]) >>> batch = pa.RecordBatch.from_arrays([n_legs, animals], ... names=["n_legs", "animals"]) >>> batch.to_pylist() [{'n_legs': 2, 'animals': 'Flamingo'}, {'n_legs': 2, ...}, {'n_legs': 100, 'animals': 'Centipede'}]
- to_string(self, show_metadata=False)¶
- validate(self, *, full=False)¶
Perform validation checks. An exception is raised if validation fails.
By default only cheap validation checks are run. Pass full=True for thorough validation checks (potentially O(n)).