pyarrow.ChunkedArray¶
- class pyarrow.ChunkedArray¶
Bases:
_PandasConvertible
An array-like composed from a (possibly empty) collection of pyarrow.Arrays
Warning
Do not call this class’s constructor directly.
Examples
To construct a ChunkedArray object use
pyarrow.chunked_array()
:>>> import pyarrow as pa >>> pa.chunked_array([], type=pa.int8()) <pyarrow.lib.ChunkedArray object at ...> [ ... ]
>>> pa.chunked_array([[2, 2, 4], [4, 5, 100]]) <pyarrow.lib.ChunkedArray object at ...> [ [ 2, 2, 4 ], [ 4, 5, 100 ] ] >>> isinstance(pa.chunked_array([[2, 2, 4], [4, 5, 100]]), pa.ChunkedArray) True
- __init__(*args, **kwargs)¶
Methods
__init__
(*args, **kwargs)cast
(self[, target_type, safe, options])Cast array values to another data type
chunk
(self, i)Select a chunk by its index.
combine_chunks
(self, MemoryPool memory_pool=None)Flatten this ChunkedArray into a single non-chunked array.
dictionary_encode
(self[, null_encoding])Compute dictionary-encoded representation of array.
drop_null
(self)Remove missing values from a chunked array.
equals
(self, ChunkedArray other)Return whether the contents of two chunked arrays are equal.
fill_null
(self, fill_value)Replace each null element in values with fill_value.
filter
(self, mask[, null_selection_behavior])Select values from the chunked array.
flatten
(self, MemoryPool memory_pool=None)Flatten this ChunkedArray.
format
(self, **kwargs)get_total_buffer_size
(self)The sum of bytes in each buffer referenced by the chunked array.
index
(self, value[, start, end, memory_pool])Find the first index of a value.
is_null
(self, *[, nan_is_null])Return boolean array indicating the null values.
is_valid
(self)Return boolean array indicating the non-null values.
iterchunks
(self)Convert to an iterator of ChunkArrays.
length
(self)Return length of a ChunkedArray.
slice
(self[, offset, length])Compute zero-copy slice of this ChunkedArray
take
(self, indices)Select values from the chunked array.
to_numpy
(self)Return a NumPy copy of this array (experimental).
to_pandas
(self[, memory_pool, categories, ...])Convert to a pandas-compatible NumPy array or DataFrame, as appropriate
to_pylist
(self)Convert to a list of native Python objects.
to_string
(self, *, int indent=0, ...)Render a "pretty-printed" string representation of the ChunkedArray
unify_dictionaries
(self, ...)Unify dictionaries across all chunks.
unique
(self)Compute distinct elements in array
validate
(self, *[, full])Perform validation checks.
value_counts
(self)Compute counts of unique elements in array.
Attributes
Convert to a list of single-chunked arrays.
Total number of bytes consumed by the elements of the chunked array.
Number of null entries
Number of underlying chunks.
Return data type of a ChunkedArray.
- cast(self, target_type=None, safe=None, options=None)¶
Cast array values to another data type
See
pyarrow.compute.cast()
for usage.- Parameters:
- Returns:
- cast
Array
orChunkedArray
- cast
Examples
>>> import pyarrow as pa >>> n_legs = pa.chunked_array([[2, 2, 4], [4, 5, 100]]) >>> n_legs.type DataType(int64)
Change the data type of an array:
>>> n_legs_seconds = n_legs.cast(pa.duration('s')) >>> n_legs_seconds.type DurationType(duration[s])
- chunk(self, i)¶
Select a chunk by its index.
- Parameters:
- i
int
- i
- Returns:
Examples
>>> import pyarrow as pa >>> n_legs = pa.chunked_array([[2, 2, None], [4, 5, 100]]) >>> n_legs.chunk(1) <pyarrow.lib.Int64Array object at ...> [ 4, 5, 100 ]
- chunks¶
Convert to a list of single-chunked arrays.
Examples
>>> import pyarrow as pa >>> n_legs = pa.chunked_array([[2, 2, None], [4, 5, 100]]) >>> n_legs <pyarrow.lib.ChunkedArray object at ...> [ [ 2, 2, null ], [ 4, 5, 100 ] ] >>> n_legs.chunks [<pyarrow.lib.Int64Array object at ...> [ 2, 2, null ], <pyarrow.lib.Int64Array object at ...> [ 4, 5, 100 ]]
- combine_chunks(self, MemoryPool memory_pool=None)¶
Flatten this ChunkedArray into a single non-chunked array.
- Parameters:
- memory_pool
MemoryPool
, defaultNone
For memory allocations, if required, otherwise use default pool
- memory_pool
- Returns:
- result
Array
- result
Examples
>>> import pyarrow as pa >>> n_legs = pa.chunked_array([[2, 2, 4], [4, 5, 100]]) >>> n_legs <pyarrow.lib.ChunkedArray object at ...> [ [ 2, 2, 4 ], [ 4, 5, 100 ] ] >>> n_legs.combine_chunks() <pyarrow.lib.Int64Array object at ...> [ 2, 2, 4, 4, 5, 100 ]
- data¶
- dictionary_encode(self, null_encoding='mask')¶
Compute dictionary-encoded representation of array.
See
pyarrow.compute.dictionary_encode()
for full usage.- Parameters:
- null_encoding
How to handle null entries.
- Returns:
- encoded
ChunkedArray
A dictionary-encoded version of this array.
- encoded
Examples
>>> import pyarrow as pa >>> animals = pa.chunked_array(( ... ["Flamingo", "Parot", "Dog"], ... ["Horse", "Brittle stars", "Centipede"] ... )) >>> animals.dictionary_encode() <pyarrow.lib.ChunkedArray object at ...> [ ... -- dictionary: [ "Flamingo", "Parot", "Dog", "Horse", "Brittle stars", "Centipede" ] -- indices: [ 0, 1, 2 ], ... -- dictionary: [ "Flamingo", "Parot", "Dog", "Horse", "Brittle stars", "Centipede" ] -- indices: [ 3, 4, 5 ] ]
- drop_null(self)¶
Remove missing values from a chunked array. See
pyarrow.compute.drop_null()
for full description.Examples
>>> import pyarrow as pa >>> n_legs = pa.chunked_array([[2, 2, None], [4, 5, 100]]) >>> n_legs <pyarrow.lib.ChunkedArray object at ...> [ [ 2, 2, null ], [ 4, 5, 100 ] ] >>> n_legs.drop_null() <pyarrow.lib.ChunkedArray object at ...> [ [ 2, 2 ], [ 4, 5, 100 ] ]
- equals(self, ChunkedArray other)¶
Return whether the contents of two chunked arrays are equal.
- Parameters:
- other
pyarrow.ChunkedArray
Chunked array to compare against.
- other
- Returns:
- are_equalbool
Examples
>>> import pyarrow as pa >>> n_legs = pa.chunked_array([[2, 2, 4], [4, 5, 100]]) >>> animals = pa.chunked_array(( ... ["Flamingo", "Parot", "Dog"], ... ["Horse", "Brittle stars", "Centipede"] ... )) >>> n_legs.equals(n_legs) True >>> n_legs.equals(animals) False
- fill_null(self, fill_value)¶
Replace each null element in values with fill_value.
See
pyarrow.compute.fill_null()
for full usage.- Parameters:
- fill_value
The replacement value for null entries.
- Returns:
- result
Array
orChunkedArray
A new array with nulls replaced by the given value.
- result
Examples
>>> import pyarrow as pa >>> fill_value = pa.scalar(5, type=pa.int8()) >>> n_legs = pa.chunked_array([[2, 2, 4], [4, None, 100]]) >>> n_legs.fill_null(fill_value) <pyarrow.lib.ChunkedArray object at ...> [ [ 2, 2, 4, 4, 5, 100 ] ]
- filter(self, mask, null_selection_behavior='drop')¶
Select values from the chunked array.
See
pyarrow.compute.filter()
for full usage.- Parameters:
- mask
Array
orarray-like
The boolean mask to filter the chunked array with.
- null_selection_behavior
How nulls in the mask should be handled.
- mask
- Returns:
- filtered
Array
orChunkedArray
An array of the same type, with only the elements selected by the boolean mask.
- filtered
Examples
>>> import pyarrow as pa >>> n_legs = pa.chunked_array([[2, 2, 4], [4, 5, 100]]) >>> n_legs <pyarrow.lib.ChunkedArray object at ...> [ [ 2, 2, 4 ], [ 4, 5, 100 ] ] >>> mask = pa.array([True, False, None, True, False, True]) >>> n_legs.filter(mask) <pyarrow.lib.ChunkedArray object at ...> [ [ 2 ], [ 4, 100 ] ] >>> n_legs.filter(mask, null_selection_behavior="emit_null") <pyarrow.lib.ChunkedArray object at ...> [ [ 2, null ], [ 4, 100 ] ]
- flatten(self, MemoryPool memory_pool=None)¶
Flatten this ChunkedArray. If it has a struct type, the column is flattened into one array per struct field.
- Parameters:
- memory_pool
MemoryPool
, defaultNone
For memory allocations, if required, otherwise use default pool
- memory_pool
- Returns:
- result
list
ofChunkedArray
- result
Examples
>>> import pyarrow as pa >>> n_legs = pa.chunked_array([[2, 2, 4], [4, 5, 100]]) >>> c_arr = pa.chunked_array(n_legs.value_counts()) >>> c_arr <pyarrow.lib.ChunkedArray object at ...> [ -- is_valid: all not null -- child 0 type: int64 [ 2, 4, 5, 100 ] -- child 1 type: int64 [ 2, 2, 1, 1 ] ] >>> c_arr.flatten() [<pyarrow.lib.ChunkedArray object at ...> [ [ 2, 4, 5, 100 ] ], <pyarrow.lib.ChunkedArray object at ...> [ [ 2, 2, 1, 1 ] ]] >>> c_arr.type StructType(struct<values: int64, counts: int64>) >>> n_legs.type DataType(int64)
- format(self, **kwargs)¶
- get_total_buffer_size(self)¶
The sum of bytes in each buffer referenced by the chunked array.
An array may only reference a portion of a buffer. This method will overestimate in this case and return the byte size of the entire buffer.
If a buffer is referenced multiple times then it will only be counted once.
Examples
>>> import pyarrow as pa >>> n_legs = pa.chunked_array([[2, 2, 4], [4, None, 100]]) >>> n_legs.get_total_buffer_size() 49
- index(self, value, start=None, end=None, *, memory_pool=None)¶
Find the first index of a value.
See
pyarrow.compute.index()
for full usage.- Parameters:
- value
Scalar
or object The value to look for in the array.
- start
int
, optional The start index where to look for value.
- end
int
, optional The end index where to look for value.
- memory_pool
MemoryPool
, optional A memory pool for potential memory allocations.
- value
- Returns:
- index
Int64Scalar
The index of the value in the array (-1 if not found).
- index
Examples
>>> import pyarrow as pa >>> n_legs = pa.chunked_array([[2, 2, 4], [4, 5, 100]]) >>> n_legs <pyarrow.lib.ChunkedArray object at ...> [ [ 2, 2, 4 ], [ 4, 5, 100 ] ] >>> n_legs.index(4) <pyarrow.Int64Scalar: 2> >>> n_legs.index(4, start=3) <pyarrow.Int64Scalar: 3>
- is_null(self, *, nan_is_null=False)¶
Return boolean array indicating the null values.
- Parameters:
- Returns:
- arraybool
Array
orChunkedArray
- arraybool
Examples
>>> import pyarrow as pa >>> n_legs = pa.chunked_array([[2, 2, 4], [4, None, 100]]) >>> n_legs.is_null() <pyarrow.lib.ChunkedArray object at ...> [ [ false, false, false, false, true, false ] ]
- is_valid(self)¶
Return boolean array indicating the non-null values.
Examples
>>> import pyarrow as pa >>> n_legs = pa.chunked_array([[2, 2, 4], [4, None, 100]]) >>> n_legs.is_valid() <pyarrow.lib.ChunkedArray object at ...> [ [ true, true, true ], [ true, false, true ] ]
- iterchunks(self)¶
Convert to an iterator of ChunkArrays.
Examples
>>> import pyarrow as pa >>> n_legs = pa.chunked_array([[2, 2, 4], [4, None, 100]]) >>> for i in n_legs.iterchunks(): ... print(i.null_count) ... 0 1
- length(self)¶
Return length of a ChunkedArray.
Examples
>>> import pyarrow as pa >>> n_legs = pa.chunked_array([[2, 2, 4], [4, 5, 100]]) >>> n_legs.length() 6
- nbytes¶
Total number of bytes consumed by the elements of the chunked array.
In other words, the sum of bytes from all buffer ranges referenced.
Unlike get_total_buffer_size this method will account for array offsets.
If buffers are shared between arrays then the shared portion will only be counted multiple times.
The dictionary of dictionary arrays will always be counted in their entirety even if the array only references a portion of the dictionary.
Examples
>>> import pyarrow as pa >>> n_legs = pa.chunked_array([[2, 2, 4], [4, None, 100]]) >>> n_legs.nbytes 49
- null_count¶
Number of null entries
- Returns:
Examples
>>> import pyarrow as pa >>> n_legs = pa.chunked_array([[2, 2, 4], [4, None, 100]]) >>> n_legs.null_count 1
- num_chunks¶
Number of underlying chunks.
- Returns:
Examples
>>> import pyarrow as pa >>> n_legs = pa.chunked_array([[2, 2, None], [4, 5, 100]]) >>> n_legs.num_chunks 2
- slice(self, offset=0, length=None)¶
Compute zero-copy slice of this ChunkedArray
- Parameters:
- Returns:
- sliced
ChunkedArray
- sliced
Examples
>>> import pyarrow as pa >>> n_legs = pa.chunked_array([[2, 2, 4], [4, 5, 100]]) >>> n_legs <pyarrow.lib.ChunkedArray object at ...> [ [ 2, 2, 4 ], [ 4, 5, 100 ] ] >>> n_legs.slice(2,2) <pyarrow.lib.ChunkedArray object at ...> [ [ 4 ], [ 4 ] ]
- take(self, indices)¶
Select values from the chunked array.
See
pyarrow.compute.take()
for full usage.- Parameters:
- indices
Array
orarray-like
The indices in the array whose values will be returned.
- indices
- Returns:
- taken
Array
orChunkedArray
An array with the same datatype, containing the taken values.
- taken
Examples
>>> import pyarrow as pa >>> n_legs = pa.chunked_array([[2, 2, 4], [4, 5, 100]]) >>> n_legs <pyarrow.lib.ChunkedArray object at ...> [ [ 2, 2, 4 ], [ 4, 5, 100 ] ] >>> n_legs.take([1,4,5]) <pyarrow.lib.ChunkedArray object at ...> [ [ 2, 5, 100 ] ]
- to_numpy(self)¶
Return a NumPy copy of this array (experimental).
- Returns:
- array
numpy.ndarray
- array
Examples
>>> import pyarrow as pa >>> n_legs = pa.chunked_array([[2, 2, 4], [4, 5, 100]]) >>> n_legs.to_numpy() array([ 2, 2, 4, 4, 5, 100])
- to_pandas(self, memory_pool=None, categories=None, bool strings_to_categorical=False, bool zero_copy_only=False, bool integer_object_nulls=False, bool date_as_object=True, bool timestamp_as_object=False, bool use_threads=True, bool deduplicate_objects=True, bool ignore_metadata=False, bool safe=True, bool split_blocks=False, bool self_destruct=False, types_mapper=None)¶
Convert to a pandas-compatible NumPy array or DataFrame, as appropriate
- Parameters:
- memory_pool
MemoryPool
, defaultNone
Arrow MemoryPool to use for allocations. Uses the default memory pool is not passed.
- strings_to_categoricalbool, default
False
Encode string (UTF8) and binary types to pandas.Categorical.
- categories
list
, defaultempty
List of fields that should be returned as pandas.Categorical. Only applies to table-like data structures.
- zero_copy_onlybool, default
False
Raise an ArrowException if this function call would require copying the underlying data.
- integer_object_nullsbool, default
False
Cast integers with nulls to objects
- date_as_objectbool, default
True
Cast dates to objects. If False, convert to datetime64[ns] dtype.
- timestamp_as_objectbool, default
False
Cast non-nanosecond timestamps (np.datetime64) to objects. This is useful if you have timestamps that don’t fit in the normal date range of nanosecond timestamps (1678 CE-2262 CE). If False, all timestamps are converted to datetime64[ns] dtype.
- use_threadsbool, default
True
Whether to parallelize the conversion using multiple threads.
- deduplicate_objectsbool, default
False
Do not create multiple copies Python objects when created, to save on memory use. Conversion will be slower.
- ignore_metadatabool, default
False
If True, do not use the ‘pandas’ metadata to reconstruct the DataFrame index, if present
- safebool, default
True
For certain data types, a cast is needed in order to store the data in a pandas DataFrame or Series (e.g. timestamps are always stored as nanoseconds in pandas). This option controls whether it is a safe cast or not.
- split_blocksbool, default
False
If True, generate one internal “block” for each column when creating a pandas.DataFrame from a RecordBatch or Table. While this can temporarily reduce memory note that various pandas operations can trigger “consolidation” which may balloon memory use.
- self_destructbool, default
False
EXPERIMENTAL: If True, attempt to deallocate the originating Arrow memory while converting the Arrow object to pandas. If you use the object after calling to_pandas with this option it will crash your program.
Note that you may not see always memory usage improvements. For example, if multiple columns share an underlying allocation, memory can’t be freed until all columns are converted.
- types_mapperfunction, default
None
A function mapping a pyarrow DataType to a pandas ExtensionDtype. This can be used to override the default pandas type for conversion of built-in pyarrow types or in absence of pandas_metadata in the Table schema. The function receives a pyarrow DataType and is expected to return a pandas ExtensionDtype or
None
if the default conversion should be used for that type. If you have a dictionary mapping, you can passdict.get
as function.
- memory_pool
- Returns:
pandas.Series
orpandas.DataFrame
depending ontype
of object
Examples
>>> import pyarrow as pa >>> import pandas as pd
Convert a Table to pandas DataFrame:
>>> table = pa.table([ ... pa.array([2, 4, 5, 100]), ... pa.array(["Flamingo", "Horse", "Brittle stars", "Centipede"]) ... ], names=['n_legs', 'animals']) >>> table.to_pandas() n_legs animals 0 2 Flamingo 1 4 Horse 2 5 Brittle stars 3 100 Centipede >>> isinstance(table.to_pandas(), pd.DataFrame) True
Convert a RecordBatch to pandas DataFrame:
>>> import pyarrow as pa >>> n_legs = pa.array([2, 4, 5, 100]) >>> animals = pa.array(["Flamingo", "Horse", "Brittle stars", "Centipede"]) >>> batch = pa.record_batch([n_legs, animals], ... names=["n_legs", "animals"]) >>> batch pyarrow.RecordBatch n_legs: int64 animals: string >>> batch.to_pandas() n_legs animals 0 2 Flamingo 1 4 Horse 2 5 Brittle stars 3 100 Centipede >>> isinstance(batch.to_pandas(), pd.DataFrame) True
Convert a Chunked Array to pandas Series:
>>> import pyarrow as pa >>> n_legs = pa.chunked_array([[2, 2, 4], [4, 5, 100]]) >>> n_legs.to_pandas() 0 2 1 2 2 4 3 4 4 5 5 100 dtype: int64 >>> isinstance(n_legs.to_pandas(), pd.Series) True
- to_pylist(self)¶
Convert to a list of native Python objects.
Examples
>>> import pyarrow as pa >>> n_legs = pa.chunked_array([[2, 2, 4], [4, None, 100]]) >>> n_legs.to_pylist() [2, 2, 4, 4, None, 100]
- to_string(self, *, int indent=0, int window=5, int container_window=2, bool skip_new_lines=False)¶
Render a “pretty-printed” string representation of the ChunkedArray
- Parameters:
- indent
int
How much to indent right the content of the array, by default
0
.- window
int
How many items to preview within each chunk at the begin and end of the chunk when the chunk is bigger than the window. The other elements will be ellipsed.
- container_window
int
How many chunks to preview at the begin and end of the array when the array is bigger than the window. The other elements will be ellipsed. This setting also applies to list columns.
- skip_new_linesbool
If the array should be rendered as a single line of text or if each element should be on its own line.
- indent
Examples
>>> import pyarrow as pa >>> n_legs = pa.chunked_array([[2, 2, 4], [4, 5, 100]]) >>> n_legs.to_string(skip_new_lines=True) '[[2,2,4],[4,5,100]]'
- type¶
Return data type of a ChunkedArray.
Examples
>>> import pyarrow as pa >>> n_legs = pa.chunked_array([[2, 2, 4], [4, 5, 100]]) >>> n_legs.type DataType(int64)
- unify_dictionaries(self, MemoryPool memory_pool=None)¶
Unify dictionaries across all chunks.
This method returns an equivalent chunked array, but where all chunks share the same dictionary values. Dictionary indices are transposed accordingly.
If there are no dictionaries in the chunked array, it is returned unchanged.
- Parameters:
- memory_pool
MemoryPool
, defaultNone
For memory allocations, if required, otherwise use default pool
- memory_pool
- Returns:
- result
ChunkedArray
- result
Examples
>>> import pyarrow as pa >>> arr_1 = pa.array(["Flamingo", "Parot", "Dog"]).dictionary_encode() >>> arr_2 = pa.array(["Horse", "Brittle stars", "Centipede"]).dictionary_encode() >>> c_arr = pa.chunked_array([arr_1, arr_2]) >>> c_arr <pyarrow.lib.ChunkedArray object at ...> [ ... -- dictionary: [ "Flamingo", "Parot", "Dog" ] -- indices: [ 0, 1, 2 ], ... -- dictionary: [ "Horse", "Brittle stars", "Centipede" ] -- indices: [ 0, 1, 2 ] ] >>> c_arr.unify_dictionaries() <pyarrow.lib.ChunkedArray object at ...> [ ... -- dictionary: [ "Flamingo", "Parot", "Dog", "Horse", "Brittle stars", "Centipede" ] -- indices: [ 0, 1, 2 ], ... -- dictionary: [ "Flamingo", "Parot", "Dog", "Horse", "Brittle stars", "Centipede" ] -- indices: [ 3, 4, 5 ] ]
- unique(self)¶
Compute distinct elements in array
- Returns:
Examples
>>> import pyarrow as pa >>> n_legs = pa.chunked_array([[2, 2, 4], [4, 5, 100]]) >>> n_legs <pyarrow.lib.ChunkedArray object at ...> [ [ 2, 2, 4 ], [ 4, 5, 100 ] ] >>> n_legs.unique() <pyarrow.lib.Int64Array object at ...> [ 2, 4, 5, 100 ]
- validate(self, *, full=False)¶
Perform validation checks. An exception is raised if validation fails.
By default only cheap validation checks are run. Pass full=True for thorough validation checks (potentially O(n)).
- value_counts(self)¶
Compute counts of unique elements in array.
Examples
>>> import pyarrow as pa >>> n_legs = pa.chunked_array([[2, 2, 4], [4, 5, 100]]) >>> n_legs <pyarrow.lib.ChunkedArray object at ...> [ [ 2, 2, 4 ], [ 4, 5, 100 ] ] >>> n_legs.value_counts() <pyarrow.lib.StructArray object at ...> -- is_valid: all not null -- child 0 type: int64 [ 2, 4, 5, 100 ] -- child 1 type: int64 [ 2, 2, 1, 1 ]