Configuration Settings¶

The following configuration options can be passed to SessionConfig to control various aspects of query execution.

For applications which do not expose SessionConfig, like datafusion-cli, these options may also be set via environment variables. To construct a session with options from the environment, use SessionConfig::from_env. The name of the environment variable is the option’s key, transformed to uppercase and with periods replaced with underscores. For example, to configure datafusion.execution.batch_size you would set the DATAFUSION_EXECUTION_BATCH_SIZE environment variable. Values are parsed according to the same rules used in casts from Utf8. If the value in the environment variable cannot be cast to the type of the configuration option, the default value will be used instead and a warning emitted. Environment variables are read during SessionConfig initialisation so they must be set beforehand and will not affect running sessions.

key	default	description
datafusion.catalog.create_default_catalog_and_schema	true	Whether the default catalog and schema should be created automatically.
datafusion.catalog.default_catalog	datafusion	The default catalog name - this impacts what SQL queries use if not specified
datafusion.catalog.default_schema	public	The default schema name - this impacts what SQL queries use if not specified
datafusion.catalog.information_schema	false	Should DataFusion provide access to `information_schema` virtual tables for displaying schema information
datafusion.catalog.location	NULL	Location scanned to load tables for `default` schema
datafusion.catalog.format	NULL	Type of `TableProvider` to use when loading `default` schema
datafusion.catalog.has_header	false	If the file has a header
datafusion.execution.batch_size	8192	Default batch size while creating new batches, it’s especially useful for buffer-in-memory batches since creating tiny batches would result in too much metadata memory consumption
datafusion.execution.coalesce_batches	true	When set to true, record batches will be examined between each operator and small batches will be coalesced into larger batches. This is helpful when there are highly selective filters or joins that could produce tiny output batches. The target batch size is determined by the configuration setting
datafusion.execution.collect_statistics	false	Should DataFusion collect statistics after listing files
datafusion.execution.target_partitions	0	Number of partitions for query execution. Increasing partitions can increase concurrency. Defaults to the number of CPU cores on the system
datafusion.execution.time_zone	+00:00	The default time zone Some functions, e.g. `EXTRACT(HOUR from SOME_TIME)`, shift the underlying datetime according to this time zone, and then extract the hour
datafusion.execution.parquet.enable_page_index	false	If true, uses parquet data page level metadata (Page Index) statistics to reduce the number of rows decoded.
datafusion.execution.parquet.pruning	true	If true, the parquet reader attempts to skip entire row groups based on the predicate in the query and the metadata (min/max values) stored in the parquet file
datafusion.execution.parquet.skip_metadata	true	If true, the parquet reader skip the optional embedded metadata that may be in the file Schema. This setting can help avoid schema conflicts when querying multiple parquet files with schemas containing compatible types but different metadata
datafusion.execution.parquet.metadata_size_hint	NULL	If specified, the parquet reader will try and fetch the last `size_hint` bytes of the parquet file optimistically. If not specified, two reads are required: One read to fetch the 8-byte parquet footer and another to fetch the metadata length encoded in the footer
datafusion.execution.parquet.pushdown_filters	false	If true, filter expressions are be applied during the parquet decoding operation to reduce the number of rows decoded
datafusion.execution.parquet.reorder_filters	false	If true, filter expressions evaluated during the parquet decoding operation will be reordered heuristically to minimize the cost of evaluation. If false, the filters are applied in the same order as written in the query
datafusion.optimizer.enable_round_robin_repartition	true	When set to true, the physical plan optimizer will try to add round robin repartitioning to increase parallelism to leverage more CPU cores
datafusion.optimizer.filter_null_join_keys	false	When set to true, the optimizer will insert filters before a join between a nullable and non-nullable column to filter out nulls on the nullable side. This filter can add additional overhead when the file format does not fully support predicate push down.
datafusion.optimizer.repartition_aggregations	true	Should DataFusion repartition data using the aggregate keys to execute aggregates in parallel using the provided `target_partitions` level
datafusion.optimizer.repartition_file_min_size	10485760	Minimum total files size in bytes to perform file scan repartitioning.
datafusion.optimizer.repartition_joins	true	Should DataFusion repartition data using the join keys to execute joins in parallel using the provided `target_partitions` level
datafusion.optimizer.repartition_file_scans	true	When set to true, file groups will be repartitioned to achieve maximum parallelism. Currently supported only for Parquet format in which case multiple row groups from the same file may be read concurrently. If false then each row group is read serially, though different files may be read in parallel.
datafusion.optimizer.repartition_windows	true	Should DataFusion repartition data using the partitions keys to execute window functions in parallel using the provided `target_partitions` level
datafusion.optimizer.repartition_sorts	true	Should DataFusion execute sorts in a per-partition fashion and merge afterwards instead of coalescing first and sorting globally. With this flag is enabled, plans in the form below “SortExec: [a@0 ASC]”, “ CoalescePartitionsExec”, “ RepartitionExec: partitioning=RoundRobinBatch(8), input_partitions=1”, would turn into the plan below which performs better in multithreaded environments “SortPreservingMergeExec: [a@0 ASC]”, “ SortExec: [a@0 ASC]”, “ RepartitionExec: partitioning=RoundRobinBatch(8), input_partitions=1”,
datafusion.optimizer.skip_failed_rules	true	When set to true, the logical plan optimizer will produce warning messages if any optimization rules produce errors and then proceed to the next rule. When set to false, any rules that produce errors will cause the query to fail
datafusion.optimizer.max_passes	3	Number of times that the optimizer will attempt to optimize the plan
datafusion.optimizer.top_down_join_key_reordering	true	When set to true, the physical plan optimizer will run a top down process to reorder the join keys
datafusion.optimizer.prefer_hash_join	true	When set to true, the physical plan optimizer will prefer HashJoin over SortMergeJoin. HashJoin can work more efficiently than SortMergeJoin but consumes more memory
datafusion.optimizer.hash_join_single_partition_threshold	1048576	The maximum estimated size in bytes for one input side of a HashJoin will be collected into a single partition
datafusion.explain.logical_plan_only	false	When set to true, the explain statement will only print logical plans
datafusion.explain.physical_plan_only	false	When set to true, the explain statement will only print physical plans
datafusion.sql_parser.parse_float_as_decimal	false	When set to true, SQL parser will parse float as decimal type
datafusion.sql_parser.enable_ident_normalization	true	When set to true, SQL parser will normalize ident (convert ident to lowercase when not quoted)

Status

Frequently Asked Questions