Configuration

BallistaContext Configuration Settings

Ballista has a number of configuration settings that can be specified when creating a BallistaContext.

Example: Specifying configuration options when creating a context

let config = BallistaConfig::builder()
.set("ballista.shuffle.partitions", "200")
.set("ballista.batch.size", "16384")
.build() ?;

let ctx = BallistaContext::remote("localhost", 50050, & config).await?;

Ballista Configuration Settings

key

type

default

description

ballista.job.name

Utf8

N/A

Sets the job name that will appear in the web user interface for any submitted jobs.

ballista.shuffle.partitions

UInt16

16

Sets the default number of partitions to create when repartitioning query stages.

ballista.batch.size

UInt16

8192

Sets the default batch size.

ballista.repartition.joins

Boolean

true

When set to true, Ballista will repartition data using the join keys to execute joins in parallel using the provided ballista.shuffle.partitions level.

ballista.repartition.aggregations

Boolean

true

When set to true, Ballista will repartition data using the aggregate keys to execute aggregates in parallel using the provided ballista.shuffle.partitions level.

ballista.repartition.windows

Boolean

true

When set to true, Ballista will repartition data using the partition keys to execute window functions in parallel using the provided ballista.shuffle.partitions level.

ballista.parquet.pruning

Boolean

true

Determines whether Parquet pruning should be enabled or not.

ballista.with_information_schema

Boolean

true

Determines whether the information_schema should be created in the context. This is necessary for supporting DDL commands such as SHOW TABLES.

ballista.plugin_dir

Boolean

true

Specified a path for plugin files. Dynamic library files in this directory will be loaded when scheduler state initializes.

DataFusion Configuration Settings

In addition to Ballista-specific configuration settings, the following DataFusion settings can also be specified.

key

type

default

description

datafusion.execution.coalesce_batches

Boolean

true

When set to true, record batches will be examined between each operator and small batches will be coalesced into larger batches. This is helpful when there are highly selective filters or joins that could produce tiny output batches. The target batch size is determined by the configuration setting ‘datafusion.execution.coalesce_target_batch_size’.

datafusion.execution.coalesce_target_batch_size

UInt64

4096

Target batch size when coalescing batches. Uses in conjunction with the configuration setting ‘datafusion.execution.coalesce_batches’.

datafusion.explain.logical_plan_only

Boolean

false

When set to true, the explain statement will only print logical plans.

datafusion.explain.physical_plan_only

Boolean

false

When set to true, the explain statement will only print physical plans.

datafusion.optimizer.filter_null_join_keys

Boolean

false

When set to true, the optimizer will insert filters before a join between a nullable and non-nullable column to filter out nulls on the nullable side. This filter can add additional overhead when the file format does not fully support predicate push down.

datafusion.optimizer.skip_failed_rules

Boolean

true

When set to true, the logical plan optimizer will produce warning messages if any optimization rules produce errors and then proceed to the next rule. When set to false, any rules that produce errors will cause the query to fail.

Ballista Scheduler Configuration Settings

Besides the BallistaContext configuration settings, a few configuration settings for the Ballista scheduler to better manage the whole cluster are also needed to be taken care of.

Example: Specifying configuration options when starting the scheduler

./ballista-scheduler --scheduler-policy push-staged --event-loop-buffer-size 1000000 --executor-slots-policy
round-robin-local

key

type

default

description

scheduler-policy

Utf8

pull-staged

Sets the task scheduling policy for the scheduler, possible values: pull-staged, push-staged.

event-loop-buffer-size

UInt32

10000

Sets the event loop buffer size. for a system of high throughput, a larger value like 1000000 is recommended.

executor-slots-policy

Utf8

bias

Sets the executor slots policy for the scheduler, possible values: bias, round-robin, round-robin-local. For a cluster with single scheduler, round-robin-local is recommended.

finished-job-data-clean-up-interval-seconds

UInt64

300

Sets the delayed interval for cleaning up finished job data, mainly the shuffle data, 0 means the cleaning up is disabled.

finished-job-state-clean-up-interval-seconds

UInt64

3600

Sets the delayed interval for cleaning up finished job state stored in the backend, 0 means the cleaning up is disabled.

advertise-flight-sql-endpoint

Utf8

N/A

Sets the route endpoint for proxying flight sql results via scheduler.