Building Arrow C++#
System setup#
Arrow uses CMake as a build configuration system. We recommend building out-of-source. If you are not familiar with this terminology:
In-source build:
cmake
is invoked directly from thecpp
directory. This can be inflexible when you wish to maintain multiple build environments (e.g. one for debug builds and another for release builds)Out-of-source build:
cmake
is invoked from another directory, creating an isolated build environment that does not interact with any other build environment. For example, you could createcpp/build-debug
and invokecmake $CMAKE_ARGS ..
from this directory
Building requires:
A C++17-enabled compiler. On Linux, gcc 7.1 and higher should be sufficient. For Windows, at least Visual Studio VS2017 is required.
CMake 3.16 or higher
On Linux and macOS, either
make
orninja
build utilitiesAt least 1GB of RAM for a minimal build, 4GB for a minimal debug build with tests and 8GB for a full build using docker.
On Ubuntu/Debian you can install the requirements with:
sudo apt-get install \
build-essential \
ninja-build \
cmake
On Alpine Linux:
apk add autoconf \
bash \
cmake \
g++ \
gcc \
ninja \
make
On Fedora Linux:
sudo dnf install \
cmake \
gcc \
gcc-c++ \
ninja-build \
make
On Arch Linux:
sudo pacman -S --needed \
base-devel \
ninja \
cmake
On macOS, you can use Homebrew:
git clone https://github.com/apache/arrow.git
cd arrow
brew update && brew bundle --file=cpp/Brewfile
With vcpkg:
git clone https://github.com/apache/arrow.git
cd arrow
vcpkg install \
--x-manifest-root cpp \
--feature-flags=versions \
--clean-after-build
On MSYS2:
pacman --sync --refresh --noconfirm \
ccache \
git \
mingw-w64-${MSYSTEM_CARCH}-boost \
mingw-w64-${MSYSTEM_CARCH}-brotli \
mingw-w64-${MSYSTEM_CARCH}-cmake \
mingw-w64-${MSYSTEM_CARCH}-gcc \
mingw-w64-${MSYSTEM_CARCH}-gflags \
mingw-w64-${MSYSTEM_CARCH}-glog \
mingw-w64-${MSYSTEM_CARCH}-gtest \
mingw-w64-${MSYSTEM_CARCH}-lz4 \
mingw-w64-${MSYSTEM_CARCH}-protobuf \
mingw-w64-${MSYSTEM_CARCH}-python3-numpy \
mingw-w64-${MSYSTEM_CARCH}-rapidjson \
mingw-w64-${MSYSTEM_CARCH}-snappy \
mingw-w64-${MSYSTEM_CARCH}-thrift \
mingw-w64-${MSYSTEM_CARCH}-zlib \
mingw-w64-${MSYSTEM_CARCH}-zstd
Building#
All the instructions below assume that you have cloned the Arrow git
repository and navigated to the cpp
subdirectory:
$ git clone https://github.com/apache/arrow.git
$ cd arrow/cpp
CMake presets#
Using CMake version 3.21.0 or higher, some presets for various build
configurations are provided. You can get a list of the available presets
using cmake --list-presets
:
$ cmake --list-presets # from inside the `cpp` subdirectory
Available configure presets:
"ninja-debug-minimal" - Debug build without anything enabled
"ninja-debug-basic" - Debug build with tests and reduced dependencies
"ninja-debug" - Debug build with tests and more optional components
[ etc. ]
You can inspect the specific options enabled by a given preset using
cmake -N --preset <preset name>
:
$ cmake --preset -N ninja-debug-minimal
Preset CMake variables:
ARROW_BUILD_INTEGRATION="OFF"
ARROW_BUILD_STATIC="OFF"
ARROW_BUILD_TESTS="OFF"
ARROW_EXTRA_ERROR_CONTEXT="ON"
ARROW_WITH_RE2="OFF"
ARROW_WITH_UTF8PROC="OFF"
CMAKE_BUILD_TYPE="Debug"
You can also create a build from a given preset:
$ mkdir build # from inside the `cpp` subdirectory
$ cd build
$ cmake .. --preset ninja-debug-minimal
Preset CMake variables:
ARROW_BUILD_INTEGRATION="OFF"
ARROW_BUILD_STATIC="OFF"
ARROW_BUILD_TESTS="OFF"
ARROW_EXTRA_ERROR_CONTEXT="ON"
ARROW_WITH_RE2="OFF"
ARROW_WITH_UTF8PROC="OFF"
CMAKE_BUILD_TYPE="Debug"
-- Building using CMake version: 3.21.3
[ etc. ]
and then ask to compile the build targets:
$ cmake --build .
[142/142] Creating library symlink debug/libarrow.so.700 debug/libarrow.so
$ tree debug/
debug/
├── libarrow.so -> libarrow.so.700
├── libarrow.so.700 -> libarrow.so.700.0.0
└── libarrow.so.700.0.0
0 directories, 3 files
When creating a build, it is possible to pass custom options besides the preset-defined ones, for example:
$ cmake .. --preset ninja-debug-minimal -DCMAKE_INSTALL_PREFIX=/usr/local
Note
The CMake presets are provided as a help to get started with Arrow development and understand common build configurations. They are not guaranteed to be immutable but may change in the future based on feedback.
Instead of relying on CMake presets, it is highly recommended that automated builds, continuous integration, release scripts, etc. use manual configuration, as outlined below.
See also
Manual configuration#
The build system uses CMAKE_BUILD_TYPE=release
by default, so if this
argument is omitted then a release build will be produced.
Note
You need to set more options to build on Windows. See Developing on Windows for details.
Several build types are possible:
Debug
: doesn’t apply any compiler optimizations and adds debugging information in the binary.RelWithDebInfo
: applies compiler optimizations while adding debug information in the binary.Release
: applies compiler optimizations and removes debug information from the binary.
Note
These build types provide suitable optimization/debug flags by
default but you can change them by specifying
-DARROW_C_FLAGS_${BUILD_TYPE}=...
and/or
-DARROW_CXX_FLAGS_${BUILD_TYPE}=...
. ${BUILD_TYPE}
is upper
case of build type. For example, DEBUG
(-DARROW_C_FLAGS_DEBUG=...
/ -DARROW_CXX_FLAGS_DEBUG=...
) for the
Debug
build type and RELWITHDEBINFO
(-DARROW_C_FLAGS_RELWITHDEBINFO=...
/
-DARROW_CXX_FLAGS_RELWITHDEBINFO=...
) for the RelWithDebInfo
build type.
For example, you can use -O3
as an optimization flag for the Release
build type by passing -DARROW_CXX_FLAGS_RELEASE=-O3
.
You can use -g3
as a debug flag for the Debug
build type
by passing -DARROW_CXX_FLAGS_DEBUG=-g3
.
You can also use the standard CMAKE_C_FLAGS_${BUILD_TYPE}
and CMAKE_CXX_FLAGS_${BUILD_TYPE}
variables but
the ARROW_C_FLAGS_${BUILD_TYPE}
and
ARROW_CXX_FLAGS_${BUILD_TYPE}
variables are
recommended. The CMAKE_C_FLAGS_${BUILD_TYPE}
and
CMAKE_CXX_FLAGS_${BUILD_TYPE}
variables replace all default
flags provided by CMake, while ARROW_C_FLAGS_${BUILD_TYPE}
and
ARROW_CXX_FLAGS_${BUILD_TYPE}
just append the
flags specified, which allows selectively overriding some of the defaults.
You can also run default build with flag -DARROW_EXTRA_ERROR_CONTEXT=ON
, see
Extra debugging help.
Minimal release build (1GB of RAM for building or more recommended):
$ mkdir build-release
$ cd build-release
$ cmake ..
$ make -j8 # if you have 8 CPU cores, otherwise adjust
Minimal debug build with unit tests (4GB of RAM for building or more recommended):
$ git submodule update --init --recursive
$ export ARROW_TEST_DATA=$PWD/../testing/data
$ mkdir build-debug
$ cd build-debug
$ cmake -DCMAKE_BUILD_TYPE=Debug -DARROW_BUILD_TESTS=ON ..
$ make -j8 # if you have 8 CPU cores, otherwise adjust
$ make unittest # to run the tests
The unit tests are not built by default. After building, one can also invoke
the unit tests using the ctest
tool provided by CMake (note that test
depends on python
being available).
On some Linux distributions, running the test suite might require setting an
explicit locale. If you see any locale-related errors, try setting the
environment variable (which requires the locales
package or equivalent):
$ export LC_ALL="en_US.UTF-8"
Faster builds with Ninja#
Many contributors use the Ninja build system to
get faster builds. It especially speeds up incremental builds. To use
ninja
, pass -GNinja
when calling cmake
and then use the ninja
command instead of make
.
Unity builds#
The CMake
unity builds
option can make full builds significantly faster, but it also increases the
memory requirements. Consider turning it on (using -DCMAKE_UNITY_BUILD=ON
)
if memory consumption is not an issue.
Optional Components#
By default, the C++ build system creates a fairly minimal build. We have
several optional system components which you can opt into building by passing
boolean flags to cmake
.
-DARROW_BUILD_UTILITIES=ON
: Build Arrow commandline utilities-DARROW_COMPUTE=ON
: Build all computational kernel functions-DARROW_CSV=ON
: CSV reader module-DARROW_CUDA=ON
: CUDA integration for GPU development. Depends on NVIDIA CUDA toolkit. The CUDA toolchain used to build the library can be customized by using the$CUDA_HOME
environment variable.-DARROW_DATASET=ON
: Dataset API, implies the Filesystem API-DARROW_FILESYSTEM=ON
: Filesystem API for accessing local and remote filesystems-DARROW_FLIGHT=ON
: Arrow Flight RPC system, which depends at least on gRPC-DARROW_FLIGHT_SQL=ON
: Arrow Flight SQL-DARROW_GANDIVA=ON
: Gandiva expression compiler, depends on LLVM, Protocol Buffers, and re2-DARROW_GANDIVA_JAVA=ON
: Gandiva JNI bindings for Java-DARROW_GCS=ON
: Build Arrow with GCS support (requires the GCloud SDK for C++)-DARROW_HDFS=ON
: Arrow integration with libhdfs for accessing the Hadoop Filesystem-DARROW_JEMALLOC=ON
: Build the Arrow jemalloc-based allocator, on by default-DARROW_JSON=ON
: JSON reader module-DARROW_MIMALLOC=ON
: Build the Arrow mimalloc-based allocator-DARROW_ORC=ON
: Arrow integration with Apache ORC-DARROW_PARQUET=ON
: Apache Parquet libraries and Arrow integration-DPARQUET_REQUIRE_ENCRYPTION=ON
: Parquet Modular Encryption-DARROW_PYTHON=ON
: This option is deprecated since 10.0.0. This will be removed in a future release. Use CMake presets instead. Or you can enableARROW_COMPUTE
,ARROW_CSV
,ARROW_DATASET
,ARROW_FILESYSTEM
,ARROW_HDFS
, andARROW_JSON
directly instead.-DARROW_S3=ON
: Support for Amazon S3-compatible filesystems-DARROW_SUBSTRAIT=ON
: Build with support for Substrait-DARROW_WITH_RE2=ON
: Build with support for regular expressions using the re2 library, on by default and used whenARROW_COMPUTE
orARROW_GANDIVA
isON
-DARROW_WITH_UTF8PROC=ON
: Build with support for Unicode properties using the utf8proc library, on by default and used whenARROW_COMPUTE
orARROW_GANDIVA
isON
-DARROW_TENSORFLOW=ON
: Build Arrow with TensorFlow support enabled
Compression options available in Arrow are:
-DARROW_WITH_BROTLI=ON
: Build support for Brotli compression-DARROW_WITH_BZ2=ON
: Build support for BZ2 compression-DARROW_WITH_LZ4=ON
: Build support for lz4 compression-DARROW_WITH_SNAPPY=ON
: Build support for Snappy compression-DARROW_WITH_ZLIB=ON
: Build support for zlib (gzip) compression-DARROW_WITH_ZSTD=ON
: Build support for ZSTD compression
Some features of the core Arrow shared library can be switched off for improved build times if they are not required for your application:
-DARROW_IPC=ON
: build the IPC extensions
Note
If your use-case is limited to reading/writing Arrow data then the default
options should be sufficient. However, if you wish to build any tests/benchmarks
then ARROW_JSON
is also required (it will be enabled automatically).
If extended format support is desired then adding ARROW_PARQUET
, ARROW_CSV
,
ARROW_JSON
, or ARROW_ORC
shouldn’t enable any additional components.
Note
In general, it’s a good idea to enable ARROW_COMPUTE
if you anticipate using
any compute kernels beyond cast
. While there are (as of 12.0.0) a handful of
additional kernels built in by default, this list may change in the future as it’s
partly based on kernel usage in the current format implementations.
Optional Targets#
For development builds, you will often want to enable additional targets in
enable to exercise your changes, using the following cmake
options.
-DARROW_BUILD_BENCHMARKS=ON
: Build executable benchmarks.-DARROW_BUILD_EXAMPLES=ON
: Build examples of using the Arrow C++ API.-DARROW_BUILD_INTEGRATION=ON
: Build additional executables that are used to exercise protocol interoperability between the different Arrow implementations.-DARROW_BUILD_UTILITIES=ON
: Build executable utilities.-DARROW_BUILD_TESTS=ON
: Build executable unit tests.-DARROW_ENABLE_TIMING_TESTS=ON
: If building unit tests, enable those unit tests that rely on wall-clock timing (this flag is disabled on CI because it can make test results flaky).-DARROW_FUZZING=ON
: Build fuzz targets and related executables.
Optional Checks#
The following special checks are available as well. They instrument the generated code in various ways so as to detect select classes of problems at runtime (for example when executing unit tests).
-DARROW_USE_ASAN=ON
: Enable Address Sanitizer to check for memory leaks, buffer overflows or other kinds of memory management issues.-DARROW_USE_TSAN=ON
: Enable Thread Sanitizer to check for races in multi-threaded code.-DARROW_USE_UBSAN=ON
: Enable Undefined Behavior Sanitizer to check for situations which trigger C++ undefined behavior.
Some of those options are mutually incompatible, so you may have to build several times with different options if you want to exercise all of them.
CMake version requirements#
We support CMake 3.16 and higher.
LLVM and Clang Tools#
We are currently using LLVM for library builds and for other developer tools
such as code formatting with clang-format
. LLVM can be installed via most
modern package managers (apt, yum, conda, Homebrew, vcpkg, chocolatey).
Build Dependency Management#
The build system supports a number of third-party dependencies
AWSSDK
: for S3 support, requires system cURL and can use theBUNDLED
method described below
benchmark
: Google benchmark, for testing
Boost
: for cross-platform support
Brotli
: for data compression
BZip2
: for data compression
c-ares
: a dependency of gRPC
gflags
: for command line utilities (formerly Googleflags)
GLOG
: for logging
google_cloud_cpp_storage
: for Google Cloud Storage support, requires system cURL and can use theBUNDLED
method described below
gRPC
: for remote procedure calls
GTest
: Googletest, for testing
LLVM
: a dependency of Gandiva
Lz4
: for data compression
ORC
: for Apache ORC format support
re2
: for compute kernels and Gandiva, a dependency of gRPC
Protobuf
: Google Protocol Buffers, for data serialization
RapidJSON
: for data serialization
Snappy
: for data compression
Thrift
: Apache Thrift, for data serialization
utf8proc
: for compute kernels
ZLIB
: for data compression
zstd
: for data compression
The CMake option ARROW_DEPENDENCY_SOURCE
is a global option that instructs
the build system how to resolve each dependency. There are a few options:
AUTO
: Try to find package in the system default locations and build from source if not foundBUNDLED
: Building the dependency automatically from sourceSYSTEM
: Finding the dependency in system paths using CMake’s built-infind_package
function, or usingpkg-config
for packages that do not have this featureCONDA
: Use$CONDA_PREFIX
as alternativeSYSTEM
PATHVCPKG
: Find dependencies installed by vcpkg, and if not found, runvcpkg install
to install themBREW
: Use Homebrew default paths as an alternativeSYSTEM
path
The default method is AUTO
unless you are developing within an active conda
environment (detected by presence of the $CONDA_PREFIX
environment
variable), in which case it is CONDA
.
Individual Dependency Resolution#
While -DARROW_DEPENDENCY_SOURCE=$SOURCE
sets a global default for all
packages, the resolution strategy can be overridden for individual packages by
setting -D$PACKAGE_NAME_SOURCE=..
. For example, to build Protocol Buffers
from source, set
-DProtobuf_SOURCE=BUNDLED
This variable is unfortunately case-sensitive; the name used for each package is listed above, but the most up-to-date listing can be found in cpp/cmake_modules/ThirdpartyToolchain.cmake.
Bundled Dependency Versions#
When using the BUNDLED
method to build a dependency from source, the
version number from cpp/thirdparty/versions.txt
is used. There is also a
dependency source downloader script (see below), which can be used to set up
offline builds.
When using BUNDLED
for dependency resolution (and if you use either the
jemalloc or mimalloc allocators, which are recommended), statically linking the
Arrow libraries in a third party project is more complex. See below for
instructions about how to configure your build system in this case.
Offline Builds#
If you do not use the above variables to direct the Arrow build system to preinstalled dependencies, they will be built automatically by the Arrow build system. The source archive for each dependency will be downloaded via the internet, which can cause issues in environments with limited access to the internet.
To enable offline builds, you can download the source artifacts yourself and
use environment variables of the form ARROW_$LIBRARY_URL
to direct the
build system to read from a local file rather than accessing the internet.
To make this easier for you, we have prepared a script
thirdparty/download_dependencies.sh
which will download the correct version
of each dependency to a directory of your choosing. It will print a list of
bash-style environment variable statements at the end to use for your build
script.
# Download tarballs into $HOME/arrow-thirdparty
$ ./thirdparty/download_dependencies.sh $HOME/arrow-thirdparty
You can then invoke CMake to create the build directory and it will use the declared environment variable pointing to downloaded archives instead of downloading them (one for each build dir!).
Statically Linking#
When -DARROW_BUILD_STATIC=ON
, all build dependencies built as static
libraries by the Arrow build system will be merged together to create a static
library arrow_bundled_dependencies
. In UNIX-like environments (Linux, macOS,
MinGW), this is called libarrow_bundled_dependencies.a
and on Windows with
Visual Studio arrow_bundled_dependencies.lib
. This “dependency bundle”
library is installed in the same place as the other Arrow static libraries.
If you are using CMake, the bundled dependencies will automatically be included
when linking if you use the arrow_static
CMake target. In other build
systems, you may need to explicitly link to the dependency bundle. We created
an example CMake-based build configuration to
show you a working example.
On Linux and macOS, if your application does not link to the pthread
library already, you must include -pthread
in your linker setup. In CMake
this can be accomplished with the Threads
built-in package:
set(THREADS_PREFER_PTHREAD_FLAG ON)
find_package(Threads REQUIRED)
target_link_libraries(my_target PRIVATE Threads::Threads)
Extra debugging help#
If you use the CMake option -DARROW_EXTRA_ERROR_CONTEXT=ON
it will compile
the libraries with extra debugging information on error checks inside the
RETURN_NOT_OK
macro. In unit tests with ASSERT_OK
, this will yield error
outputs like:
../src/arrow/ipc/ipc-read-write-test.cc:609: Failure
Failed
../src/arrow/ipc/metadata-internal.cc:508 code: TypeToFlatbuffer(fbb, *field.type(), &children, &layout, &type_enum, dictionary_memo, &type_offset)
../src/arrow/ipc/metadata-internal.cc:598 code: FieldToFlatbuffer(fbb, *schema.field(i), dictionary_memo, &offset)
../src/arrow/ipc/metadata-internal.cc:651 code: SchemaToFlatbuffer(fbb, schema, dictionary_memo, &fb_schema)
../src/arrow/ipc/writer.cc:697 code: WriteSchemaMessage(schema_, dictionary_memo_, &schema_fb)
../src/arrow/ipc/writer.cc:730 code: WriteSchema()
../src/arrow/ipc/writer.cc:755 code: schema_writer.Write(&dictionaries_)
../src/arrow/ipc/writer.cc:778 code: CheckStarted()
../src/arrow/ipc/ipc-read-write-test.cc:574 code: writer->WriteRecordBatch(batch)
NotImplemented: Unable to convert type: decimal(19, 4)
Deprecations and API Changes#
We use the marco ARROW_DEPRECATED
which wraps C++ deprecated attribute for
APIs that have been deprecated. It is a good practice to compile third party
applications with -Werror=deprecated-declarations
(for GCC/Clang or similar
flags of other compilers) to proactively catch and account for API changes.
Modular Build Targets#
Since there are several major parts of the C++ project, we have provided modular CMake targets for building each library component, group of unit tests and benchmarks, and their dependencies:
make arrow
for Arrow core librariesmake parquet
for Parquet librariesmake gandiva
for Gandiva (LLVM expression compiler) libraries
Note
If you have selected Ninja as CMake generator, replace make arrow
with
ninja arrow
, and so on.
To build the unit tests or benchmarks, add -tests
or -benchmarks
to the target name. So make arrow-tests
will build the Arrow core unit
tests. Using the -all
target, e.g. parquet-all
, will build everything.
If you wish to only build and install one or more project subcomponents, we
have provided the CMake option ARROW_OPTIONAL_INSTALL
to only install
targets that have been built. For example, if you only wish to build the
Parquet libraries, its tests, and its dependencies, you can run:
cmake .. -DARROW_PARQUET=ON \
-DARROW_OPTIONAL_INSTALL=ON \
-DARROW_BUILD_TESTS=ON
make parquet
make install
If you omit an explicit target when invoking make
, all targets will be
built.
Debugging with Xcode on macOS#
Xcode is the IDE provided with macOS and can be use to develop and debug Arrow by generating an Xcode project:
cd cpp
mkdir xcode-build
cd xcode-build
cmake .. -G Xcode -DARROW_BUILD_TESTS=ON -DCMAKE_BUILD_TYPE=DEBUG
open arrow.xcodeproj
This will generate a project and open it in the Xcode app. As an alternative,
the command xcodebuild
will perform a command-line build using the
generated project. It is recommended to use the “Automatically Create Schemes”
option when first launching the project. Selecting an auto-generated scheme
will allow you to build and run a unittest with breakpoints enabled.