Starting a Ballista Cluster using Docker Compose

Docker Compose is a convenient way to launch a cluster when testing locally.

Build Docker Images

There are no officially published Docker images, so it is currently necessary to build the images from source. Run the following commands to clone the source repository and build the Docker image.

git clone -b 0.12.0
cd arrow-ballista

This will create the following images:

  • apache/arrow-ballista-benchmarks:0.12.0

  • apache/arrow-ballista-cli:0.12.0

  • apache/arrow-ballista-executor:0.12.0

  • apache/arrow-ballista-scheduler:0.12.0

  • apache/arrow-ballista-standalone:0.12.0

Start a Cluster

Using the docker-compose.yml from the source repository, run the following command to start a cluster:

docker-compose up --build

This should show output similar to the following:

$ docker-compose up
Creating network "ballista-benchmarks_default" with the default driver
Creating ballista-benchmarks_etcd_1 ... done
Creating ballista-benchmarks_ballista-scheduler_1 ... done
Creating ballista-benchmarks_ballista-executor_1  ... done
Attaching to ballista-benchmarks_etcd_1, ballista-benchmarks_ballista-scheduler_1, ballista-benchmarks_ballista-executor_1
ballista-executor_1   | [2021-08-28T15:55:22Z INFO  ballista_executor] Running with config:
ballista-executor_1   | [2021-08-28T15:55:22Z INFO  ballista_executor] work_dir: /tmp/.tmpLVx39c
ballista-executor_1   | [2021-08-28T15:55:22Z INFO  ballista_executor] concurrent_tasks: 4
ballista-scheduler_1  | [2021-08-28T15:55:22Z INFO  ballista_scheduler] Ballista v0.12.0 Scheduler listening on
ballista-executor_1   | [2021-08-28T15:55:22Z INFO  ballista_executor] Ballista v0.12.0 Rust Executor listening on

The scheduler listens on port 50050 and this is the port that clients will need to connect to.

The scheduler web UI is available on port 80 in the scheduler.

Connect from the Ballista CLI

docker run --network=host -it apache/arrow-ballista-cli:0.12.0 --host localhost --port 50050