pyarrow.dataset.FilenamePartitioning#

class pyarrow.dataset.FilenamePartitioning(Schema schema, dictionaries=None, segment_encoding=u'uri')#

Bases: pyarrow._dataset.KeyValuePartitioning

A Partitioning based on a specified Schema.

The FilenamePartitioning expects one segment in the file name for each field in the schema (all fields are required to be present) separated by ‘_’. For example given schema<year:int16, month:int8> the name "2009_11_" would be parsed to (“year” == 2009 and “month” == 11).

Parameters
schemaSchema

The schema that describes the partitions present in the file path.

dictionariesdict[str, Array]

If the type of any field of schema is a dictionary type, the corresponding entry of dictionaries must be an array containing every value which may be taken by the corresponding column or an error will be raised in parsing.

segment_encodingstr, default “uri”

After splitting paths into segments, decode the segments. Valid values are “uri” (URI-decode segments) and “none” (leave as-is).

Returns
FilenamePartitioning

Examples

>>> from pyarrow.dataset import FilenamePartitioning
>>> partitioning = FilenamePartitioning(
...     pa.schema([("year", pa.int16()), ("month", pa.int8())]))
>>> print(partitioning.parse("2009_11_"))
((year == 2009:int16) and (month == 11:int8))
__init__(*args, **kwargs)#

Methods

__init__(*args, **kwargs)

discover([field_names, infer_dictionary, ...])

Discover a FilenamePartitioning.

parse(self, path)

Attributes

dictionaries

The unique values for each partition field, if available.

schema

The arrow Schema attached to the partitioning.
dictionaries#

The unique values for each partition field, if available.

Those values are only available if the Partitioning object was created through dataset discovery from a PartitioningFactory, or if the dictionaries were manually specified in the constructor. If no dictionary field is available, this returns an empty list.

static discover(field_names=None, infer_dictionary=False, schema=None, segment_encoding='uri')#

Discover a FilenamePartitioning.

Parameters
field_nameslist of str

The names to associate with the values from the subdirectory names. If schema is given, will be populated from the schema.

infer_dictionarybool, default False

When inferring a schema for partition fields, yield dictionary encoded types instead of plain types. This can be more efficient when materializing virtual columns, and Expressions parsed by the finished Partitioning will include dictionaries of all unique inspected values for each field.

schemaSchema, default None

Use this schema instead of inferring a schema from partition values. Partition values will be validated against this schema before accumulation into the Partitioning’s dictionary.

segment_encodingstr, default “uri”

After splitting paths into segments, decode the segments. Valid values are “uri” (URI-decode segments) and “none” (leave as-is).

Returns
PartitioningFactory

To be used in the FileSystemFactoryOptions.

parse(self, path)#
schema#

The arrow Schema attached to the partitioning.

previous

pyarrow.dataset.HivePartitioning

next

pyarrow.dataset.Dataset