Release Management Guide¶
This page provides detailed information on the steps followed to perform a release. It can be used both as a guide to learn the ADBC release process and as a comprehensive checklist for the Release Manager when performing a release.
Principles¶
The Apache Arrow Release follows the guidelines defined at the Apache Software Foundation Release Policy.
Preparing for the release¶
Some steps of the release require being a committer or a PMC member.
A GPG key in the Apache Web of Trust to sign artifacts. This will have to be cross signed by other Apache committers/PMC members. You must set your GPG key ID in
dev/release/.env
(seedev/release/.env.example
for a template).The GPG key needs to be added to this SVN repo and this one.
Configure Maven to publish artifacts to Apache repositories. You will need to setup a master password at
~/.m2/settings-security.xml
andsettings.xml
as specified on the Apache guide. It can be tested with the following command:# You might need to export GPG_TTY=$(tty) to properly prompt for a passphrase mvn clean install -Papache-release
An Artifactory API key (log in with your ASF credentials, then generate it from your profile in the upper-right). You must set the Artifactory API key in
dev/release/.env
(seedev/release/.env.example
for a template).Install
en_US.UTF-8
locale. You can confirm available locales bylocale -a
.Install Conda with conda-forge, and create and activate the environment.
mamba create -n adbc -c conda-forge --file ci/conda_env_dev.txt
This will install two tools used in the release process:
commitizen
(generates changelog from commit messages) andgh
(submit jobs/download artifacts).Install Docker.
Clone the main Arrow repository (https://github.com/apache/arrow) and symlink
arrow-adbc/dev/release/.env
toarrow/dev/release/.env
. Some release scripts depend on the scripts in the main Arrow repository.
Before creating a Release Candidate¶
Regenerate the LICENSE.txt (see CONTRIBUTING.md) and create a pull request if any changes were needed.
# Setup gpg agent for signing artifacts
source dev/release/setup-gpg-agent.sh
# Activate conda environment
mamba activate adbc
Check Nightly Verification Job¶
Ensure that the verification job is passing. This simulates part of the release verification workflow to detect issues ahead of time.
Creating a Release Candidate¶
These are the different steps that are required to create a Release Candidate.
For the initial Release Candidate, we will create a maintenance branch from master. Follow up Release Candidates will update the maintenance branch by cherry-picking specific commits.
We have implemented a Feature Freeze policy between Release Candidates. This means that, in general, we should only add bug fixes between Release Candidates. In rare cases, critical features can be added between Release Candidates, if there is community consensus.
Create or update the corresponding maintenance branch¶
# Execute the following from an up to date master branch.
# This will create a branch locally called maint-X.Y.Z.
# X.Y.Z corresponds with the Major, Minor and Patch version number
# of the release respectively. As an example 9.0.0
git branch maint-X.Y.Z
# Push the maintenance branch to the remote repository
git push -u apache maint-X.Y.Z
git switch maint-X.Y.Z
# Remove the commits that created the changelog and bumped the
# versions, since 01-prepare.sh will redo those steps
git reset --hard HEAD~2
# Cherry-pick any commits by hand.
git cherry-pick ...
# Push the updated maintenance branch to the remote repository
git push -u apache maint-X.Y.Z
Create the Release Candidate tag from the updated maintenance branch¶
# Start from the updated maintenance branch.
git switch maint-X.Y.Z
# The following script will create a branch for the Release Candidate,
# place the necessary commits updating the version number and changelog, and then create a git tag
# on OSX use gnu-sed with homebrew: brew install gnu-sed (and export to $PATH)
#
# <rc-number> starts at 0 and increments every time the Release Candidate is burned
# so for the first RC this would be: dev/release/01-prepare.sh 1.0.0 0
dev/release/01-prepare.sh <arrow-dir> <rc-number>
git push -u apache apache-arrow-adbc-<release>-rc<rc-number> maint-<release>
Build source and binaries and submit them¶
# Download the produced source and binaries, sign them, and add the
# signatures to the GitHub release
#
# On macOS the only way I could get this to work was running "echo
# "UPDATESTARTUPTTY" | gpg-connect-agent" before running this
# comment otherwise I got errors referencing "ioctl" errors.
dev/release/02-sign.sh <rc-number>
# Upload the source release tarball and signs to
# https://dist.apache.org/repos/dist/dev/arrow .
dev/release/03-source.sh <rc-number>
# Upload the Java artifacts
#
# Note that you need to press the "Close" button manually by Web interface
# after you complete the script:
# https://repository.apache.org/#stagingRepositories
dev/release/04-java-upload.sh <arrow-dir> <rc-number>
# Sign and upload the deb/rpm packages and APT/Yum repositories
#
# This reuses release scripts in apache/arrow. So you need to
# specify cloned apache/arrow directory.
dev/release/05-linux-upload.sh <arrow-dir> <rc-number>
# Start verifications for binaries and wheels
dev/release/06-binary-verify.sh <rc-number>
Verify the Release¶
Start the vote thread on dev@arrow.apache.org using the template email from 06-binary-verify.sh
.
Voting and approval¶
Start the vote thread on dev@arrow.apache.org and supply instructions for verifying the integrity of the release. Approval requires a net of 3 +1 votes from PMC members. A release cannot be vetoed.
How to Verify Release Candidates¶
Install dependencies. At minimum, you will need:
cURL
Docker (to verify binaries)
Git
GnuPG
shasum (built into macOS) or sha256sum/sha512sum (on Linux)
You will also need to install all dependencies to build and verify all languages. Roughly, this means:
C and C++ compilers (or the equivalent of
build-essential
for your platform)Python 3
- Ruby with headers
meson is required
bundler, rake, red-arrow, and test-unit Ruby gems
- GLib and gobject-introspection with headers
pkg-config or cmake must be able to find libarrow-glib.so
GI_TYPELIB_PATH should be set to the path to the girepository-1.0 directory
- Java JRE and JDK (Java 8+)
the javadoc command must also be accessible
Go
CMake, ninja-build, libpq (with headers), SQLite (with headers)
Alternatively, you can have the verification script download and install dependencies automatically via Conda. See the environment variables below.
Clone the project:
$ git clone https://github.com/apache/arrow-adbc.git
Run the verification script:
$ cd arrow-adbc # Pass the release and the RC number $ ./dev/release/verify-release-candidate.sh 0.1.0 6
These environment variables may be helpful:
ARROW_TMPDIR=/path/to/directory
to specify the temporary directory used. Using a fixed directory can help avoid repeating the same setup and build steps if the script has to be run multiple times.USE_CONDA=1
to download and set up Conda for dependencies. In this case, fewer dependencies are required from the system. (Git, GnuPG, cURL, and some others are still required.)
Once finished and once the script passes, reply to the mailing list vote thread with a +1 or a -1.
Post-release tasks¶
After the release vote, we must undertake many tasks to update source artifacts, binary builds, and the Arrow website.
Be sure to go through on the following checklist:
Close the GitHub milestone/project
Open https://github.com/orgs/apache/projects and find the project
Click “…” for the project
Select “Close”
Open https://github.com/apache/arrow-adbc/milestones and find the milestone
Click “Close”
Add the new release to the Apache Reporter System
Add relevant release data for Arrow to Apache reporter.
Upload source release artifacts to Subversion
A PMC member must commit the source release artifacts to Subversion:
# dev/release/post-01-upload.sh 0
dev/release/post-01-upload.sh <rc-number>
git push apache apache-arrow-adbc-<release>
Create the final GitHub release
A committer must create the final GitHub release:
# dev/release/post-02-binary.sh 0
dev/release/post-02-binary.sh <rc-number>
Update website
This is done automatically when the tags are pushed. Please check that the nightly-website.yml workflow succeeded.
Upload wheels/sdist to PyPI
We use the twine tool to upload wheels to PyPI:
dev/release/post-03-python.sh
Publish Maven packages
Logon to the Apache repository: https://repository.apache.org/#stagingRepositories
Select the Arrow staging repository you created for RC:
orgapachearrow-XXXX
Click the
release
button
Update tags for Go modules
# dev/release/post-04-go.sh
dev/release/post-04-go.sh
Deploy APT/Yum repositories
# This reuses release scripts in apache/arrow. So you need to
# specify cloned apache/arrow directory.
#
# dev/release/post-05-linux.sh ../arrow 0
dev/release/post-05-linux.sh <arrow-dir> <rc-number>
Update R packages
This is a manual process. See the process for the Arrow R packages.
Upload Ruby packages to RubyGems
You must be one of owners of https://rubygems.org/gems/red-adbc . If you aren’t an owner of red-adbc yet, an existing owner must run the following command line to add you to red-adbc owners:
gem owner -a ${RUBYGEMS_ORG_ACCOUNT_FOR_RELEASE_MANAGER} red-adbc
An owner of red-adbc can upload:
# dev/release/post-06-ruby.sh
dev/release/post-06-ruby.sh
Upload C#/.NET packages to NuGet
You must be one of owners of the package. If you aren’t an owner yet, an existing owner can add you at https://nuget.org.
You will need to [create an API key](https://learn.microsoft.com/en-us/nuget/nuget-org/publish-a-package#create-an-api-key).
An owner can upload:
export NUGET_API_KEY=<your API key here>
# dev/release/post-07-csharp.sh
dev/release/post-07-csharp.sh
Upload Rust crates to crates.io
You must be one of owners of the package. If you aren’t an owner yet, an existing owner can add you at https://crates.io.
You will need to [create an API token](https://crates.io/settings/tokens).
An owner can upload:
cargo login
# dev/release/post-08-rust.sh
dev/release/post-08-rust.sh
Update conda-forge packages
File a PR that bumps the version to the feedstock: https://github.com/conda-forge/arrow-adbc-split-feedstock
A conda-forge or feedstock maintainer can review and merge.
Announce the new release
Write a release announcement and send to announce@apache.org and dev@arrow.apache.org.
The announcement to announce@apache.org must be sent from your apache.org e-mail address to be accepted.
Template:
dev/release/post-09-announce.sh
Remove old artifacts
Remove RC artifacts on https://dist.apache.org/repos/dist/dev/arrow/ and old release artifacts on https://dist.apache.org/repos/dist/release/arrow to follow the ASF policy:
dev/release/post-10-remove-old-artifacts.sh
Bump versions
First, update the version numbers in dev/release/versions.env
. Then, run this script to apply those version numbers to the versions embedded in files and filenames. The script will also update the changelog to the newly released changelog.
# dev/release/post-11-bump-versions.sh ../arrow
dev/release/post-11-bump-versions.sh <arrow-dir>
Publish release blog post
Run the script to generate the blog post outline, then fill out the outline and create a PR on apache/arrow-site.
# dev/release/post-12-website.sh ../arrow-site
dev/release/post-12-website.sh <arrow-site-dir>