pyarrow.parquet.write_to_dataset

pyarrow.parquet.write_to_dataset(table, root_path, partition_cols=None, partition_filename_cb=None, filesystem=None, use_legacy_dataset=True, **kwargs)[source]

Wrapper around parquet.write_table for writing a Table to Parquet format by partitions. For each combination of partition columns and values, a subdirectories are created in the following manner:

root_dir/
group1=value1
group2=value1

<uuid>.parquet

group2=value2

<uuid>.parquet

group1=valueN
group2=value1

<uuid>.parquet

group2=valueN

<uuid>.parquet

Parameters
  • table (pyarrow.Table) –

  • root_path (str, pathlib.Path) – The root directory of the dataset

  • filesystem (FileSystem, default None) – If nothing passed, paths assumed to be found in the local on-disk filesystem

  • partition_cols (list,) – Column names by which to partition the dataset Columns are partitioned in the order they are given

  • partition_filename_cb (callable,) – A callback function that takes the partition key(s) as an argument and allow you to override the partition filename. If nothing is passed, the filename will consist of a uuid.

  • use_legacy_dataset (bool, default True) – Set to False to enable the new code path (experimental, using the new Arrow Dataset API). This is more efficient when using partition columns, but does not (yet) support partition_filename_cb and metadata_collector keywords.

  • **kwargs (dict,) – Additional kwargs for write_table function. See docstring for write_table or ParquetWriter for more information. Using metadata_collector in kwargs allows one to collect the file metadata instances of dataset pieces. The file paths in the ColumnChunkMetaData will be set relative to root_path.