Release Management Guide

This page provides detailed information on the steps followed to perform a release. It can be used both as a guide to learn the ADBC release process and as a comprehensive checklist for the Release Manager when performing a release.

Principles

The Apache Arrow Release follows the guidelines defined at the Apache Software Foundation Release Policy.

Preparing for the release

Some steps of the release require being a committer or a PMC member.

  • A GPG key in the Apache Web of Trust to sign artifacts. This will have to be cross signed by other Apache committers/PMC members. You must set your GPG key ID in dev/release/.env (see dev/release/.env.example for a template).

  • The GPG key needs to be added to this SVN repo and this one.

  • Configure Maven to publish artifacts to Apache repositories. You will need to setup a master password at ~/.m2/settings-security.xml and settings.xml as specified on the Apache guide. It can be tested with the following command:

    # You might need to export GPG_TTY=$(tty) to properly prompt for a passphrase
    mvn clean install -Papache-release
    
  • An Artifactory API key (log in with your ASF credentials, then generate it from your profile in the upper-right). You must set the Artifactory API key in dev/release/.env (see dev/release/.env.example for a template).

  • Install en_US.UTF-8 locale. You can confirm available locales by locale -a.

  • Install Conda with conda-forge, and create and activate the environment.

    mamba create -n adbc -c conda-forge --file ci/conda_env_dev.txt
    

    This will install two tools used in the release process: commitizen (generates changelog from commit messages) and gh (submit jobs/download artifacts).

  • Install Docker.

  • Clone the main Arrow repository (https://github.com/apache/arrow) and symlink arrow-adbc/dev/release/.env to arrow/dev/release/.env. Some release scripts depend on the scripts in the main Arrow repository.

Before creating a Release Candidate

Regenerate the LICENSE.txt (see CONTRIBUTING.md) and create a pull request if any changes were needed.

# Setup gpg agent for signing artifacts
source dev/release/setup-gpg-agent.sh

# Activate conda environment
mamba activate adbc

Check Nightly Verification Job

Ensure that the verification job is passing. This simulates part of the release verification workflow to detect issues ahead of time.

Creating a Release Candidate

These are the different steps that are required to create a Release Candidate.

For the initial Release Candidate, we will create a maintenance branch from master. Follow up Release Candidates will update the maintenance branch by cherry-picking specific commits.

We have implemented a Feature Freeze policy between Release Candidates. This means that, in general, we should only add bug fixes between Release Candidates. In rare cases, critical features can be added between Release Candidates, if there is community consensus.

Create or update the corresponding maintenance branch

# Execute the following from an up to date master branch.
# This will create a branch locally called maint-X.Y.Z.
# X.Y.Z corresponds with the Major, Minor and Patch version number
# of the release respectively. As an example 9.0.0
git branch maint-X.Y.Z
# Push the maintenance branch to the remote repository
git push -u apache maint-X.Y.Z
git switch maint-X.Y.Z
# Remove the commits that created the changelog and bumped the
# versions, since 01-prepare.sh will redo those steps
git reset --hard HEAD~2
# Cherry-pick any commits by hand.
git cherry-pick ...
# Push the updated maintenance branch to the remote repository
git push -u apache maint-X.Y.Z

Create the Release Candidate tag from the updated maintenance branch

# Start from the updated maintenance branch.
git switch maint-X.Y.Z

# The following script will create a branch for the Release Candidate,
# place the necessary commits updating the version number and changelog, and then create a git tag
# on OSX use gnu-sed with homebrew: brew install gnu-sed (and export to $PATH)
#
# <rc-number> starts at 0 and increments every time the Release Candidate is burned
# so for the first RC this would be: dev/release/01-prepare.sh 1.0.0 0

dev/release/01-prepare.sh <arrow-dir> <rc-number>

git push -u apache apache-arrow-adbc-<release>-rc<rc-number> maint-<release>

Build source and binaries and submit them

# Download the produced source and binaries, sign them, and add the
# signatures to the GitHub release
#
# On macOS the only way I could get this to work was running "echo
# "UPDATESTARTUPTTY" | gpg-connect-agent" before running this
# comment otherwise I got errors referencing "ioctl" errors.
dev/release/02-sign.sh <rc-number>

# Upload the source release tarball and signs to
# https://dist.apache.org/repos/dist/dev/arrow .
dev/release/03-source.sh <rc-number>

# Upload the Java artifacts
#
# Note that you need to press the "Close" button manually by Web interface
# after you complete the script:
#   https://repository.apache.org/#stagingRepositories
dev/release/04-java-upload.sh <arrow-dir> <rc-number>

# Sign and upload the deb/rpm packages and APT/Yum repositories
#
# This reuses release scripts in apache/arrow. So you need to
# specify cloned apache/arrow directory.
dev/release/05-linux-upload.sh <arrow-dir> <rc-number>

# Start verifications for binaries and wheels
dev/release/06-binary-verify.sh <rc-number>

Verify the Release

Start the vote thread on dev@arrow.apache.org using the template email from 06-binary-verify.sh.

Voting and approval

Start the vote thread on dev@arrow.apache.org and supply instructions for verifying the integrity of the release. Approval requires a net of 3 +1 votes from PMC members. A release cannot be vetoed.

How to Verify Release Candidates

  1. Install dependencies. At minimum, you will need:

    • cURL

    • Docker (to verify binaries)

    • Git

    • GnuPG

    • shasum (built into macOS) or sha256sum/sha512sum (on Linux)

    You will also need to install all dependencies to build and verify all languages. Roughly, this means:

    • C and C++ compilers (or the equivalent of build-essential for your platform)

    • Python 3

    • Ruby with headers
      • meson is required

    • bundler, rake, red-arrow, and test-unit Ruby gems

    • GLib and gobject-introspection with headers
      • pkg-config or cmake must be able to find libarrow-glib.so

      • GI_TYPELIB_PATH should be set to the path to the girepository-1.0 directory

    • Java JRE and JDK (Java 8+)
      • the javadoc command must also be accessible

    • Go

    • CMake, ninja-build, libpq (with headers), SQLite (with headers)

    Alternatively, you can have the verification script download and install dependencies automatically via Conda. See the environment variables below.

  2. Clone the project:

    $ git clone https://github.com/apache/arrow-adbc.git
    
  3. Run the verification script:

    $ cd arrow-adbc
    # Pass the release and the RC number
    $ ./dev/release/verify-release-candidate.sh 0.1.0 6
    

    These environment variables may be helpful:

    • ARROW_TMPDIR=/path/to/directory to specify the temporary directory used. Using a fixed directory can help avoid repeating the same setup and build steps if the script has to be run multiple times.

    • USE_CONDA=1 to download and set up Conda for dependencies. In this case, fewer dependencies are required from the system. (Git, GnuPG, cURL, and some others are still required.)

  4. Once finished and once the script passes, reply to the mailing list vote thread with a +1 or a -1.

Post-release tasks

After the release vote, we must undertake many tasks to update source artifacts, binary builds, and the Arrow website.

Be sure to go through on the following checklist:

Close the GitHub milestone/project
Add the new release to the Apache Reporter System

Add relevant release data for Arrow to Apache reporter.

Upload source release artifacts to Subversion

A PMC member must commit the source release artifacts to Subversion:

# dev/release/post-01-upload.sh 0
dev/release/post-01-upload.sh <rc-number>
git push apache apache-arrow-adbc-<release>
Create the final GitHub release

A committer must create the final GitHub release:

# dev/release/post-02-binary.sh 0
dev/release/post-02-binary.sh <rc-number>
Update website

This is done automatically when the tags are pushed. Please check that the nightly-website.yml workflow succeeded.

Upload wheels/sdist to PyPI

We use the twine tool to upload wheels to PyPI:

dev/release/post-03-python.sh
Publish Maven packages
Update tags for Go modules
# dev/release/post-04-go.sh
dev/release/post-04-go.sh
Deploy APT/Yum repositories
# This reuses release scripts in apache/arrow. So you need to
# specify cloned apache/arrow directory.
#
# dev/release/post-05-linux.sh ../arrow 0
dev/release/post-05-linux.sh <arrow-dir> <rc-number>
Update R packages

This is a manual process. See the process for the Arrow R packages.

Upload Ruby packages to RubyGems

You must be one of owners of https://rubygems.org/gems/red-adbc . If you aren’t an owner of red-adbc yet, an existing owner must run the following command line to add you to red-adbc owners:

gem owner -a ${RUBYGEMS_ORG_ACCOUNT_FOR_RELEASE_MANAGER} red-adbc

An owner of red-adbc can upload:

# dev/release/post-06-ruby.sh
dev/release/post-06-ruby.sh
Upload C#/.NET packages to NuGet

You must be one of owners of the package. If you aren’t an owner yet, an existing owner can add you at https://nuget.org.

You will need to [create an API key](https://learn.microsoft.com/en-us/nuget/nuget-org/publish-a-package#create-an-api-key).

An owner can upload:

export NUGET_API_KEY=<your API key here>

# dev/release/post-07-csharp.sh
dev/release/post-07-csharp.sh
Upload Rust crates to crates.io

You must be one of owners of the package. If you aren’t an owner yet, an existing owner can add you at https://crates.io.

You will need to [create an API token](https://crates.io/settings/tokens).

An owner can upload:

cargo login

# dev/release/post-08-rust.sh
dev/release/post-08-rust.sh
Update conda-forge packages

File a PR that bumps the version to the feedstock: https://github.com/conda-forge/arrow-adbc-split-feedstock

A conda-forge or feedstock maintainer can review and merge.

Announce the new release

Write a release announcement and send to announce@apache.org and dev@arrow.apache.org.

The announcement to announce@apache.org must be sent from your apache.org e-mail address to be accepted.

Template:

dev/release/post-09-announce.sh
Remove old artifacts

Remove RC artifacts on https://dist.apache.org/repos/dist/dev/arrow/ and old release artifacts on https://dist.apache.org/repos/dist/release/arrow to follow the ASF policy:

dev/release/post-10-remove-old-artifacts.sh
Bump versions

First, update the version numbers in dev/release/versions.env. Then, run this script to apply those version numbers to the versions embedded in files and filenames. The script will also update the changelog to the newly released changelog.

# dev/release/post-11-bump-versions.sh ../arrow
dev/release/post-11-bump-versions.sh <arrow-dir>
Publish release blog post

Run the script to generate the blog post outline, then fill out the outline and create a PR on apache/arrow-site.

# dev/release/post-12-website.sh ../arrow-site
dev/release/post-12-website.sh <arrow-site-dir>