Release Management Guide#

This page provides detailed information on the steps followed to perform a release. It can be used both as a guide to learn the Apache Arrow release process and as a comprehensive checklist for the Release Manager when performing a release. The person acting as Release Manager must at least have committer status in order to perform the tasks below. If the Release Manager is a committer but not a member of the PMC, some tasks will need to be delegated to a PMC member and these are marked below accordingly.

Principles#

The Apache Arrow Release follows the guidelines defined at the Apache Software Foundation Release Policy.

Preparing for the release#

Before creating a source release, the Release Manager must ensure that any resolved GitHub issues have the appropriate milestone set so that the changelog is generated properly.

Note that pull requests without a corresponding GitHub issue won’t be detected by the cherry-pick script and must be cherry-picked manually by the release manager onto the maintenance branch. Examples include MINOR and Dependabot pull requests. For this reason, it’s encouraged to avoid the need for manual cherry-picking by creating issues for any pull requests that are merged to the default branch after the release maintenance branch has been created.

Requirements

Some steps of the release require being a committer or a PMC member.

  • Install the Archery utility which is required for the release.

  • You must not have any arrow-cpp or parquet-cpp environment variables defined except CC or CXX if you want to build with something other than GCC by default (e.g. clang).

  • A GPG key in the Apache Web of Trust to sign artifacts. This will have to be cross signed by other Apache committers/PMC members. If you have multiple GPG keys, you must set the correct GPG key ID in ~/.gnupg/gpg.conf by adding:

default-key ${YOUR_GPG_KEY_ID}

  • The GPG key needs to be added to this SVN repo and this one.

  • Have the build requirements for cpp and c_glib installed.

  • Set the CROSSBOW_GITHUB_TOKEN environment variable to automatically create the verify release Pull Request.

  • Install en_US.UTF-8 locale. You can confirm available locales by locale -a.

  • Install Python 3 as python

  • Create dev/release/.env from dev/release/.env.example. See the comments in dev/release/.env.example how to set each variable.

  • Setup Crossbow as defined.

  • Have Docker and Docker Compose installed.

Before creating a Release Candidate#

Ensure local tags are removed, gpg-agent is set and GitHub issues are correctly assigned.

# Delete the local tag for RC1 or later
git tag -d apache-arrow-<version>

# Setup gpg agent for signing artifacts
source dev/release/setup-gpg-agent.sh

# Curate the release
# The end of the generated report shows any GitHub issues with the wrong
# version number assigned.
archery release curate <version>

Ensure a major version milestone for a follow up release is created on GitHub. This will automatically be used by our merge script as the new version for issues closed when the maintenance branch is created.

Patch Releases#

We usually create patch releases once a major breaking issue has been identified. Issues that are identified as major breaking issues can be security fixes, broken packages for specific builds and others.

Any developer can ask for a patch release to be generated sending an email to the Arrow development mailing-list with the reason of why a new release is necessary. If there is consensus and there is a Release Manager willing to take the effort to create the release a patch release can be created.

Committers can tag issues that should be included on the next patch release using the backport-candidate label. Is the responsability of the author or the committer to add the label to the issue to help the Release Manager identify the issues that should be backported.

If a specific issue is identified as the reason to create a patch release the Release Manager should validate that, at least, this issue is correctly tagged and included in the patch release.

Be sure to go through on the following checklist:

  1. Create milestone

  2. Create maintenance branch

  3. Include issue that was requested as requiring new patch release

  4. Add new milestone to issues with backport-candidate label

  5. cherry-pick issues into maintenance branch

Creating a Release Candidate#

These are the different steps that are required to create a Release Candidate.

For the initial Release Candidate on a major release, we will create a maintenance branch from main.

Follow up Release Candidates will update the maintenance branch by cherry-picking specific commits.

For the initial Release Candidate for a minor or a patch release we will create a maintenance branch from the previous corresponding release. For example, for a 15.0.1 patch we will create a maint-15.0.1 branch from maint-15.0.0 and for a maint-15.0.2 we will create it from maint-15.0.1. Once the maintenance branch is created we will update the created maintenance branch by cherry-picking specific commits.

We have implemented a Feature Freeze policy between Release Candidates. This means that, in general, we should only add bug fixes between Release Candidates. In rare cases, critical features can be added between Release Candidates, if there is community consensus.

Create or update the corresponding maintenance branch#

# Execute the following from an up to date main branch.
# This will create a branch locally called maint-X.Y.Z.
# X.Y.Z corresponds with the Major, Minor and Patch version number
# of the release respectively. As an example 9.0.0
archery release cherry-pick X.Y.Z --execute
# Push the maintenance branch to the remote repository
git push -u apache maint-X.Y.Z

# First run in dry-mode to see which commits will be cherry-picked.
# If there are commits that we don't want to get applied, ensure the
# milestone on GitHub is set to the following release.
archery release cherry-pick X.Y.Z --continue
# Update the maintenance branch with the previous commits
archery release cherry-pick X.Y.Z --continue --execute
# Push the updated maintenance branch to the remote repository
git push -u apache maint-X.Y.Z

Optional: Test Before Creating a Release Candidate#

Some release managers prefer to perform testing before creating the first release candidate to avoid the need to create multiple release candidates within a given release.

To test before creating a release candiate:

  • Create a pull request from the up-to-date maint-X.Y.Z branch onto main

  • Title the pull request “WIP: Dummy PR to check maint-X.Y.Z status”

  • Comment on the pull request to trigger the relevant Crossbow jobs:

    • @github-actions crossbow submit --group verify-rc-source

    • @github-actions crossbow submit --group packaging

Create the Release Candidate branch from the updated maintenance branch#

# Start from the updated maintenance branch.
git checkout maint-X.Y.Z

# The following script will create a branch for the Release Candidate,
# place the necessary commits updating the version number and then create a git tag
# on OSX use gnu-sed with homebrew: brew install gnu-sed (and export to $PATH)
#
# <rc-number> starts at 0 and increments every time the Release Candidate is created
# so for the first RC this would be: dev/release/01-prepare.sh 4.0.0 5.0.0 0
dev/release/01-prepare.sh <version> <next-version> <rc-number>

# Push the release candidate tag
git push -u apache apache-arrow-<version>-rc<rc-number>
# Push the release candidate branch in order to trigger verification jobs later
git push -u apache release-<version>-rc<rc-number>

Build source and binaries and submit them#

# Build the source release tarball and create Pull Request with verification tasks
#
# NOTE: This must be run by a PMC member
# NOTE: You need to have GitHub CLI installed to run this script.
dev/release/02-source.sh <version> <rc-number>

# Submit binary tasks using crossbow, the command will output the crossbow build id
dev/release/03-binary-submit.sh <version> <rc-number>

# Wait for the crossbow jobs to finish
archery crossbow status <crossbow-build-id>

# Download the produced binaries
# This will download packages to a directory called packages/release-<version>-rc<rc-number>
dev/release/04-binary-download.sh <version> <rc-number>

# Sign and upload the binaries
#
# NOTE: This must be run by a PMC member
#
# On macOS the only way I could get this to work was running "echo "UPDATESTARTUPTTY" | gpg-connect-agent" before running this comment
# otherwise I got errors referencing "ioctl" errors.
dev/release/05-binary-upload.sh <version> <rc-number>

# Sign and upload MATLAB artifacts to the GitHub Releases area.
#
# NOTE: This must be run by a PMC member
# NOTE: You need to have GitHub CLI installed to run this script.
dev/release/06-matlab-upload.sh <version> <rc-number>

# Start verifications for binaries and wheels
dev/release/07-binary-verify.sh <version> <rc-number>

Verify the Release#

# Once the automatic verification has passed start the vote thread
# on dev@arrow.apache.org. To regenerate the email template use
SOURCE_DEFAULT=0 SOURCE_VOTE=1 dev/release/02-source.sh <version> <rc-number>

See Release Verification Process for details.

Voting and approval#

Start the vote thread on dev@arrow.apache.org and supply instructions for verifying the integrity of the release. Approval requires a net of 3 +1 votes from PMC members. A release cannot be vetoed.

Post-release tasks#

After the release vote, we must undertake many tasks to update source artifacts, binary builds, and the Arrow website.

Be sure to go through on the following checklist:

  1. Update the released milestone Date and set to “Closed” on GitHub

  2. Merge changes on release branch to maintenance branch for patch releases

  3. Add the new release to the Apache Reporter System

  4. Push release tag

  5. Upload source

  6. Upload binaries

  7. Update website

  8. Update GitHub Release Notes

  9. Update Homebrew packages

  10. Update MSYS2 package

  11. Upload RubyGems

  12. Upload JavaScript packages

  13. Upload C# packages

  14. Update conda recipes

  15. Upload wheels/sdist to pypi

  16. Update R packages

  17. Update vcpkg port

  18. Update Conan recipe

  19. Bump versions

  20. Update docs

  21. Update version in Apache Arrow Cookbook

  22. Announce the new release

  23. Publish release blog posts

  24. Announce the release on Twitter

  25. Remove old artifacts

Merge changes on release branch to maintenance branch for patch releases

Merge release-X.Y.Z-rcN to maint-X.Y.Z:

# git checkout maint-10.0.0
git checkout maint-X.Y.Z
# git merge release-10.0.0-rc0
git merge release-X.Y.Z-rcN
# git push -u apache maint-10.0.0
git push -u apache maint-X.Y.Z
Add the new release to the Apache Reporter System

Add relevant release data for Arrow to Apache reporter.

Push release tag and create GitHub Release

A committer must push the release tag to GitHub:

# dev/release/post-01-tag.sh 0.1.0 0
dev/release/post-01-tag.sh <version> <rc>
Upload source release artifacts to Subversion

A PMC member must commit the source release artifacts to Subversion:

# dev/release/post-02-upload.sh 0.1.0 0
dev/release/post-02-upload.sh <version> <rc>
Upload binary release artifacts to Artifactory

A committer must upload the binary release artifacts to Artifactory and create the GitHub Release:

# dev/release/post-03-binary.sh 0.1.0 0
dev/release/post-03-binary.sh <version> <rc number>
Update website

Add a release note for the new version to our website and update the latest release information:

## Prepare your fork of https://github.com/apache/arrow-site .
## You need to do this only once.
# git clone git@github.com:kou/arrow-site.git ../
git clone git@github.com:<YOUR_GITHUB_ID>/arrow-site.git ../
cd ../arrow-site
## Add git@github.com:apache/arrow-site.git as "apache" remote.
git remote add apache git@github.com:apache/arrow-site.git
cd -

## Generate a release note for the new version, update the
## latest release information automatically.
# dev/release/post-04-website.sh 9.0.0 10.0.0
dev/release/post-04-website.sh OLD_X.OLD_Y.OLD_Z X.Y.Z

This script pushes a release-note-X.Y.Z branch to your apache/arrow-site fork. You need to open a pull request from the release-note-X.Y.Z branch on your Web browser.

Update Release Notes in apache/arrow GitHub Release

A committer must run the following script. This has to be done once the Pull Request from the Update Website script has been merged:

# dev/release/post-05-update-gh-release-notes.sh 17.0.0
dev/release/post-05-update-gh-release-notes.sh <version>
Update Homebrew packages

Open a pull request to Homebrew:

## You need to run this on macOS or Linux that Homebrew is installed.

## Fork https://github.com/Homebrew/homebrew-core on GitHub.
## You need to do this only once.
##
## Prepare your fork of https://github.com/Homebrew/homebrew-core .
## You need to do this only once.
cd "$(brew --repository homebrew/core)"
# git remote add kou git@github.com:kou/homebrew-core.git
git remote add <YOUR_GITHUB_ID> git@github.com:<YOUR_GITHUB_ID>/homebrew-core.git
cd -

# dev/release/post-14-homebrew.sh 10.0.0 kou
dev/release/post-14-homebrew.sh X.Y.Z <YOUR_GITHUB_ID>

This script pushes a apache-arrow-X.Y.Z branch to your Homebrew/homebrew-core fork. You need to create a pull request from the apache-arrow-X.Y.Z branch with apache-arrow, apache-arrow-glib: X.Y.Z title on your Web browser.

Update MSYS2 packages

Open a pull request to MSYS2:

## Fork https://github.com/msys2/MINGW-packages on GitHub.
## You need to do this only once.
##
## Prepare your fork of https://github.com/msys2/MINGW-packages .
## You need to do this only once.
# git clone git@github.com:kou/MINGW-packages.git ../
git clone git@github.com:<YOUR_GITHUB_ID>/MINGW-packages.git ../
cd ../MINGW-packages
## Add https://github.com/msys2/MINGW-packages.git as "upstream" remote.
git remote add upstream https://github.com/msys2/MINGW-packages.git
cd -

# dev/release/post-13-msys2.sh 10.0.0 ../MINGW-packages
dev/release/post-13-msys2.sh X.Y.Z <YOUR_MINGW_PACKAGES_FORK>

This script pushes a arrow-X.Y.Z branch to your msys2/MINGW-packages fork. You need to create a pull request from the arrow-X.Y.Z branch with arrow: Update to X.Y.Z title on your Web browser.

Update RubyGems

You need an account on https://rubygems.org/ to release Ruby packages.

If you have an account on https://rubygems.org/ , you need to join owners of our gems.

Existing owners can add a new account to the owners of them by the following command line:

# dev/release/account-ruby.sh raulcd
dev/release/account-ruby.sh NEW_ACCOUNT

Update RubyGems after Homebrew packages and MSYS2 packages are updated:

# dev/release/post-06-ruby.sh 10.0.0
dev/release/post-06-ruby.sh X.Y.Z
Update JavaScript packages

In order to publish the binary build to npm, you will need to get access to the project by asking one of the current collaborators listed at https://www.npmjs.com/package/apache-arrow packages.

The package upload requires npm and yarn to be installed and 2FA to be configured on your account.

When you have access, you can publish releases to npm by running the following script:

# Login to npmjs.com (You need to do this only for the first time)
npm login --registry=https://registry.yarnpkg.com/

# dev/release/post-07-js.sh 10.0.0
dev/release/post-07-js.sh X.Y.Z
Update C# packages

You need an account on https://www.nuget.org/. You need to join owners of Apache.Arrow package. Existing owners can invite you to the owners at https://www.nuget.org/packages/Apache.Arrow/Manage .

You need to create an API key at https://www.nuget.org/account/apikeys to upload from command line.

Install the latest .NET Core SDK from https://dotnet.microsoft.com/download .

# NUGET_API_KEY=YOUR_NUGET_API_KEY dev/release/post-08-csharp.sh 10.0.0
NUGET_API_KEY=<your NuGet API key> dev/release/post-08-csharp.sh X.Y.Z
Upload wheels/sdist to PyPI

pip binary packages (called “wheels”) and source package (called “sdist”) are built using the crossbow tool that we used above during the release candidate creation process and then uploaded to PyPI (Python Package Index) under the pyarrow package.

We use the twine tool to upload wheels to PyPI:

# dev/release/post-11-python.sh 10.0.0
dev/release/post-11-python.sh <version>
Update R packages

To publish the R package on CRAN, there are a few steps we need to do first in order to ensure that binaries for Windows and macOS are available to CRAN. Jeroen Ooms <jeroenooms@gmail.com> maintains several projects that build C++ dependencies for R packages for macOS and Windows. We test copies of these same build scripts in our CI, and at release time, we need to send any changes we have and update the versions/hashes upstream.

When the release candidate is made, make draft pull requests to each repository using the rc, updating the version and SHA, as well as any cmake build changes from the corresponding files in apache/arrow. Jeroen may merge these PRs before the release vote passes, build the binary artifacts, and publish them in the right places so that we can do pre-submission checks (see below). After the release candidate vote passes, update these PRs to point to the official (non-rc) URL and mark them as ready for review. Jeroen will merge, build the binary artifacts, and publish them in the right places. See the packaging checklist. for a precise list of pull requests that must be made prior to submission to CRAN.

Once these binary prerequisites have been satisfied, we can submit to CRAN. Given the vagaries of the process, it is best if the R developers on the project verify the CRAN-worthiness of the package before submitting. Our CI systems give us some coverage for the things that CRAN checks, but there are a couple of final tests we should do to confirm that the release binaries will work and that everything runs on the same infrastructure that CRAN has, which is difficult/impossible to emulate fully with Docker. For a precise list of checks, see the packaging checklist.

Once all checks are clean, we submit to CRAN, which has a web form for uploading packages. The release process requires email confirmation from the R package maintainer, currently Neal Richardson.

Update vcpkg port

Open a pull request to vcpkg:

## Fork https://github.com/microsoft/vcpkg on GitHub.
## You need to do this only once.
##
## Prepare your fork of https://github.com/microsoft/vcpkg .
## You need to do this only once.
# git clone git@github.com:kou/vcpkg.git ../
git clone git@github.com:<YOUR_GITHUB_ID>/vcpkg.git ../
cd ../vcpkg
./bootstrap-vcpkg.sh
## Add https://github.com/microsoft/vcpkg.git as "upstream" remote.
git remote add upstream https://github.com/microsoft/vcpkg.git
cd -

# dev/release/post-15-vcpkg.sh 10.0.0 ../vcpkg
dev/release/post-15-vcpkg.sh X.Y.Z <YOUR_VCPKG_FORK>

This script pushes a arrow-X.Y.Z branch to your microsoft/vcpkg fork. You need to create a pull request from the arrow-X.Y.Z branch with [arrow] Update to X.Y.Z title on your Web browser.

Update Conan port

Open a pull request to Conan:

## Fork https://github.com/conan-io/conan-center-index on GitHub.
## You need to do this only once.
##
## Prepare your fork of https://github.com/conan-io/conan-center-index .
## You need to do this only once.
# git clone git@github.com:kou/conan-center-index.git ../
git clone git@github.com:<YOUR_GITHUB_ID>/conan-center-index.git ../
cd ../conan-center-index
## Add https://github.com/conan-io/conan-center-index.git as "upstream" remote.
git remote add upstream https://github.com/conan-io/conan-center-index.git
cd -

# dev/release/post-16-conan.sh 10.0.1 ../conan-center-index
dev/release/post-16-conan.sh X.Y.Z <YOUR_CONAN_CENTER_INDEX_FORK>

This script pushes a arrow-X.Y.Z branch to your conan-io/conan-center-index fork. You need to create a pull request from the arrow-X.Y.Z branch on your Web browser.

Bump versions
# You can run the script with BUMP_TAG=0 and BUMP_PUSH=0
# this will avoid default pushing to main and pushing the tag
# but you will require to push manually after reviewing the commits.
# dev/release/post-12-bump-versions.sh 10.0.0 11.0.0
dev/release/post-12-bump-versions.sh X.Y.Z NEXT_X.NEXT_Y.NEXT_Z
Update docs

Documentation is generated as part of the release process. We just need to upload the generated documentation:

## Prepare your fork of https://github.com/apache/arrow-site .
## You need to do this only once.
# git clone git@github.com:kou/arrow-site.git ../
git clone git@github.com:<YOUR_GITHUB_ID>/arrow-site.git ../
cd ../arrow-site
## Add git@github.com:apache/arrow-site.git as "apache" remote.
git remote add apache git@github.com:apache/arrow-site.git
cd -

# dev/release/post-10-docs.sh 10.0.0 9.0.0
dev/release/post-10-docs.sh X.Y.Z PREVIOUS_X.PREVIOUS_Y.PREVIOUS_Z

This script pushes a release-docs-X.Y.Z branch to your arrow-site fork. You need to create a Pull Request and use the asf-site branch as base for it.

Update version in Apache Arrow Cookbook

Follow the documentation in the Apache Arrow Cookbook repository

Announce the new release

Write a release announcement (see example) and send to announce@apache.org and dev@arrow.apache.org.

The announcement to announce@apache.org must be sent from your apache.org e-mail address to be accepted.

Publish release blog post

The blog post process isn’t automated. The rough set of steps we usually take are:

  • Clone apache/arrow-site.

  • Create a new branch off main for the blog post pull request we’re creating.

  • Duplicate a recent blog post entry in the _posts subfolder and update the filename and YAML metadata.

    • Set the date in the filename and in the YAML metadata to the date that the release candidate vote thread for the release closed (in GMT).

  • For minor releases only, remove any section about community updates (new committers, PMC members, etc).

  • Update the remainder of the text as needed

  • Create the pull request

  • In the pull request, ping contributors in each section requesting help filling in the details for each section.

Announce the release on social media

Post about the release and link to the blog post on social media. The project has two official accounts:

PMC members have access or can request access to post under these accounts.

Remove old artifacts

Remove RC artifacts on https://dist.apache.org/repos/dist/dev/arrow/ and old release artifacts on https://dist.apache.org/repos/dist/release/arrow to follow the ASF policy:

dev/release/post-09-remove-old-artifacts.sh

Note: This step must be done by a PMC member.