Contributing Overview¶
Local git conventions¶
If you are tracking the Arrow source repository locally, here is a
checklist for using git
:
Work off of your personal fork of
apache/arrow
and submit pull requests “upstream”.Keep your fork’s master branch synced with
upstream/master
.Develop on branches, rather than your own “master” branch.
It does not matter what you call your branch. Some people like to use the JIRA number as branch name, others use descriptive names.
Sync your branch with
upstream/master
regularly, as many commits are merged to master every day.It is recommended to use
git rebase
rather thangit merge
.In case there are conflicts, and your local commit history has multiple commits, you may simplify the conflict resolution process by squashing your local commits into a single commit. Preserving the commit history isn’t as important because when your feature branch is merged upstream, a squash happens automatically.
Pull request and review¶
When contributing a patch, use this list as a checklist of Apache Arrow workflow:
Submit the patch as a GitHub pull request against the master branch.
So that your pull request syncs with the JIRA issue, prefix your pull request name with the JIRA issue id (ex: ARROW-767: [C++] Filesystem abstraction).
Give the pull request a clear, brief description: when the pull request is merged, this will be retained in the extended commit message.
Make sure that your code passes the unit tests. You can find instructions how to run the unit tests for each Arrow component in its respective README file.
Core developers and others with a stake in the part of the project your change affects will review, request changes, and hopefully indicate their approval in the end. To make the review process smooth for everyone, try to
Break your work into small, single-purpose patches if possible.
It’s much harder to merge in a large change with a lot of disjoint features, and particularly if you’re new to the project, smaller changes are much easier for maintainers to accept.
Add new unit tests for your code.
Follow the style guides for the part(s) of the project you’re modifying.
Some languages (C++ and Python, for example) run a lint check in continuous integration. For all languages, see their respective developer documentation and READMEs for style guidance.
Try to make it look as if the codebase has a single author, and emulate any conventions you see, whether or not they are officially documented or checked.
When tests are passing and the pull request has been approved by the interested parties, a committer will merge the pull request. This is done with a command-line utility that does a squash merge.
Experimental repositories¶
Apache Arrow has an explicit policy over developing experimental repositories in the context of rules for revolutionaries.
The main motivation for this policy is to offer a lightweight mechanism to conduct experimental work, with the necessary creative freedom, within the ASF and the Apache Arrow governance model. This policy allows committers to work on new repositories, as they offer many important tools to manage it (e.g. github issues, “watch”, “github stars” to measure overall interest).
Process¶
A committer may initiate experimental work by creating a separate git repository within the Apache Arrow (e.g. via selfserve) and announcing it on the mailing list, together with its goals, and a link to the newly created repository.
The committer must initiate an email thread with the sole purpose of presenting updates to the community about the status of the repo.
There must not be official releases from the repository.
Any decision to make the experimental repo official in any way, whether by merging or migrating, must be discussed and voted on in the mailing list.
The committer is responsible for managing issues, documentation, CI of the repository, including licensing checks.
The committer decides when the repository is archived.
Repository management¶
The repository must be under
apache/
The repository’s name must be prefixed by
arrow-experimental-
The committer has full permissions over the repository (within possible in ASF)
Push / merge permissions must only be granted to Apache Arrow committers
Development process¶
The repository must follow the ASF requirements about 3rd party code.
The committer decides how to manage issues, PRs, etc.
Divergences¶
If any of the “must” above fails to materialize and no correction measure is taken by the committer upon request, the PMC should take ownership and decide what to do.
Guidance for specific features¶
From time to time the community has discussions on specific types of features and improvements that they expect to support. This section outlines decisions that have been made in this regard.
Endianness¶
The Arrow format allows setting endianness. Due to the popularity of little endian architectures most of implementation assume little endian by default. There has been some effort to support big endian platforms as well. Based on a mailing-list discussion, the requirements for a new platform are:
A robust (non-flaky, returning results in a reasonable time) Continuous Integration setup.
Benchmarks for performance critical parts of the code to demonstrate no regression.
Furthermore, for big-endian support, there are two levels that an implementation can support:
Native endianness (all Arrow communication happens with processes of the same endianness). This includes ancillary functionality such as reading and writing various file formats, such as Parquet.
Cross endian support (implementations will do byte reordering when appropriate for IPC and Flight messages).
The decision on what level to support is based on maintainers’ preferences for complexity and technical risk. In general all implementations should be open to native endianness support (provided the CI and performance requirements are met). Cross endianness support is a question for individual maintainers.
The current implementations aiming for cross endian support are:
C++
Implementations that do not intend to implement cross endian support:
Java
For other libraries, a discussion to gather consensus on the mailing-list should be had before submitting PRs.