Daily Development using Archery

To ease some of the daily development tasks, we developed a Python-written utility called Archery.

Installation

Archery requires Python 3.9 or later. It is recommended to install Archery in editable mode with the -e flag to automatically update the installation when pulling the Arrow repository. After cloning the Arrow repository, from the top level directory install Archery by using the command

$ pip install -e "dev/archery[all]"

Many operations in Archery make use of Docker and docker-compose, which you may also want to install.

Usage

You can inspect Archery usage by passing the --help flag:

$ archery --help
Usage: archery [OPTIONS] COMMAND [ARGS]...

  Apache Arrow developer utilities.

  See sub-commands help with `archery <cmd> --help`.

Options:
  --debug      Increase logging with debugging output.
  --pdb        Invoke pdb on uncaught exception.
  -q, --quiet  Silence executed commands.
  --help       Show this message and exit.

Commands:
  benchmark    Arrow benchmarking.
  build        Initialize an Arrow C++ build
  crossbow     Schedule packaging tasks or nightly builds on CI services.
  docker       Interact with docker-compose based builds.
  integration  Execute protocol and Flight integration tests
  linking      Quick and dirty utilities for checking library linkage.
  lint         Check Arrow source tree for errors
  numpydoc     Lint python docstring with NumpyDoc
  release      Release related commands.
  trigger-bot

Archery exposes independent subcommands, each of which provides dedicated help output, for example:

$ archery docker --help
Usage: archery docker [OPTIONS] COMMAND [ARGS]...

  Interact with docker-compose based builds.

Options:
  --src <arrow_src>  Specify Arrow source directory.
  --help             Show this message and exit.

Commands:
  images  List the available docker-compose images.
  push    Push the generated docker-compose image.
  run     Execute docker-compose builds.

A more detailed introduction to using Docker with Archery is available at Running Docker Builds.