pyarrow.parquet.write_table¶

pyarrow.parquet.write_table(table, where, row_group_size=None, version='1.0', use_dictionary=True, compression='snappy', write_statistics=True, use_deprecated_int96_timestamps=None, coerce_timestamps=None, allow_truncated_timestamps=False, data_page_size=None, flavor=None, filesystem=None, compression_level=None, use_byte_stream_split=False, data_page_version='1.0', use_compliant_nested_type=False, **kwargs)[source]¶

Write a Table to Parquet format.

Parameters

table (pyarrow.Table) –
where (string or pyarrow.NativeFile) –
row_group_size (int) – The number of rows per rowgroup
version ({"1.0", "2.4", "2.6"}, default "1.0") – Determine which Parquet logical types are available for use, whether the reduced set from the Parquet 1.x.x format or the expanded logical types added in later format versions. Files written with version=’2.4’ or ‘2.6’ may not be readable in all Parquet implementations, so version=’1.0’ is likely the choice that maximizes file compatibility. UINT32 and some logical types are only available with version ‘2.4’. Nanosecond timestamps are only available with version ‘2.6’. Other features such as compression algorithms or the new serialized data page format must be enabled separately (see ‘compression’ and ‘data_page_version’).
use_dictionary (bool or list) – Specify if we should use dictionary encoding in general or only for some columns.
use_deprecated_int96_timestamps (bool, default None) – Write timestamps to INT96 Parquet format. Defaults to False unless enabled by flavor argument. This take priority over the coerce_timestamps option.
coerce_timestamps (str, default None) – Cast timestamps to a particular resolution. If omitted, defaults are chosen depending on version. By default, for version='1.0' (the default) and version='2.4', nanoseconds are cast to microseconds (‘us’), while for other version values, they are written natively without loss of resolution. Seconds are always cast to milliseconds (‘ms’) by default, as Parquet does not have any temporal type with seconds resolution. If the casting results in loss of data, it will raise an exception unless allow_truncated_timestamps=True is given. Valid values: {None, ‘ms’, ‘us’}
data_page_size (int, default None) – Set a target threshold for the approximate encoded size of data pages within a column chunk (in bytes). If None, use the default data page size of 1MByte.
allow_truncated_timestamps (bool, default False) – Allow loss of data when coercing timestamps to a particular resolution. E.g. if microsecond or nanosecond data is lost when coercing to ‘ms’, do not raise an exception. Passing allow_truncated_timestamp=True will NOT result in the truncation exception being ignored unless coerce_timestamps is not None.
compression (str or dict) – Specify the compression codec, either on a general basis or per-column. Valid values: {‘NONE’, ‘SNAPPY’, ‘GZIP’, ‘BROTLI’, ‘LZ4’, ‘ZSTD’}.
write_statistics (bool or list) – Specify if we should write statistics in general (default is True) or only for some columns.
flavor ({'spark'}, default None) – Sanitize schema or set other compatibility options to work with various target systems.
filesystem (FileSystem, default None) – If nothing passed, will be inferred from where if path-like, else where is already a file-like object so no filesystem is needed.
compression_level (int or dict, default None) – Specify the compression level for a codec, either on a general basis or per-column. If None is passed, arrow selects the compression level for the compression codec in use. The compression level has a different meaning for each codec, so you have to read the documentation of the codec you are using. An exception is thrown if the compression codec does not allow specifying a compression level.
use_byte_stream_split (bool or list, default False) – Specify if the byte_stream_split encoding should be used in general or only for some columns. If both dictionary and byte_stream_stream are enabled, then dictionary is preferred. The byte_stream_split encoding is valid only for floating-point data types and should be combined with a compression codec.
data_page_version ({"1.0", "2.0"}, default "1.0") – The serialized Parquet data page format version to write, defaults to 1.0. This does not impact the file schema logical types and Arrow to Parquet type casting behavior; for that use the “version” option.
use_compliant_nested_type (bool, default False) –
Whether to write compliant Parquet nested type (lists) as defined here, defaults to False. For use_compliant_nested_type=True, this will write into a list with 3-level structure where the middle level, named list, is a repeated group with a single field named element:
```
<list-repetition> group <name> (LIST) {
    repeated group list {
          <element-repetition> <element-type> element;
    }
}
```
For use_compliant_nested_type=False, this will also write into a list with 3-level structure, where the name of the single field of the middle level list is taken from the element name for nested columns in Arrow, which defaults to item:
```
<list-repetition> group <name> (LIST) {
    repeated group list {
        <element-repetition> <element-type> item;
    }
}
```
**kwargs (optional) – Additional options for ParquetWriter

pyarrow.parquet.write_metadata

pyarrow.parquet.write_to_dataset