Developing on Windows¶
Like Linux and macOS, we have worked to enable builds to work “out of the box” with CMake for a reasonably large subset of the project.
Microsoft provides the free Visual Studio Community edition. When doing development in the shell, you must initialize the development environment each time you open the shell.
For Visual Studio 2015, execute the following batch script:
"C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\vcvarsall.bat" amd64
For Visual Studio 2017, the script is:
"C:\Program Files (x86)\Microsoft Visual Studio\2017\Community\Common7\Tools\VsDevCmd.bat" -arch=amd64
For Visual Studio 2019, the script is:
"C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\Common7\Tools\VsDevCmd.bat" -arch=amd64
One can configure a console emulator like cmder to automatically launch this when starting a new development console.
Using conda-forge for build dependencies¶
Miniconda is a minimal Python distribution including the conda package manager. Some memers of the Apache Arrow community participate in the maintenance of conda-forge, a community-maintained cross-platform package repository for conda.
conda-forge for your C++ build dependencies on Windows, first
download and install a 64-bit distribution from the Miniconda homepage
conda to use the
conda-forge channel by default, launch a
command prompt (
cmd.exe), run the initialization command shown
run the command:
conda config --add channels conda-forge
Now, you can bootstrap a build environment (call from the root directory of the Arrow codebase):
conda create -y -n arrow-dev --file=ci\conda_env_cpp.yml
Then “activate” this conda environment with:
If the environment has been activated, the Arrow build system will
automatically see the
%CONDA_PREFIX% environment variable and use that for
resolving the build dependencies. This is equivalent to setting
-DARROW_DEPENDENCY_SOURCE=SYSTEM ^ -DARROW_PACKAGE_PREFIX=%CONDA_PREFIX%\Library
To use the Visual Studio IDE with this conda environment activated, launch it by
running the command
devenv from the same command prompt.
Note that dependencies installed as conda packages are built in release mode and
cannot link with debug builds. If you intend to use
then you must build the packages from source.
-DCMAKE_BUILD_TYPE=relwithdebinfo is also available, which produces a build
that can both be linked with release libraries and be debugged.
If you run into any problems using conda packages for dependencies, a very
common problem is mixing packages from the
defaults channel with those
conda-forge. You can examine the installed packages in your
environment (and their origin) with
Using vcpkg for build dependencies¶
vcpkg is an open source package manager from Microsoft. It hosts community-contributed ports of C and C++ packages and their dependencies. Arrow includes a manifest file cpp/vcpkg.json that specifies which vcpkg packages are required to build the C++ library.
vcpkg install ^ --triplet x64-windows ^ --x-manifest-root cpp ^ --clean-after-build
On Windows, vcpkg builds dynamic link libraries by default. Use the triplet
x64-windows-static to build static libraries. vcpkg downloads source
packages and compiles them locally, so installing dependencies with vcpkg is
more time-consuming than with conda.
Then in your
cmake command, to use dependencies installed by vcpkg, set:
You can optionally set other variables to override the default CMake configurations for vcpkg, including:
-DCMAKE_TOOLCHAIN_FILE: by default, the CMake scripts automatically find the location of the vcpkg CMake toolchain file
vcpkg.cmake; use this to instead specify its location
-DVCPKG_TARGET_TRIPLET: by default, the CMake scripts attempt to infer the vcpkg triplet; use this to instead specify the triplet
-DARROW_DEPENDENCY_USE_SHARED: default is
ON; set to
OFFfor static libraries
-DVCPKG_MANIFEST_MODE: default is
ON; set to
OFFto ignore the
vcpkg.jsonmanifest file and only look for vcpkg packages that are already installed under the directory where vcpkg is installed
Building using Visual Studio (MSVC) Solution Files¶
Change working directory in
cmd.exe to the root directory of Arrow and do
an out of source build by generating a MSVC solution:
cd cpp mkdir build cd build cmake .. -G "Visual Studio 14 2015" -A x64 ^ -DARROW_BUILD_TESTS=ON cmake --build . --config Release
For newer versions of Visual Studio, specify the generator
Visual Studio 15 2017 or
Visual Studio 16 2019.
Building with Ninja and clcache¶
The Ninja build system offers better build
parallelization, and the optional clcache compiler cache keeps track of
past compilations to avoid running them over and over again (in a way similar
to the Unix-specific
Newer versions of Visual Studio include Ninja. To see if your Visual Studio
includes Ninja, run the initialization command shown
If Ninja is not included in your version of Visual Studio, and you are using conda, activate your conda environment and install Ninja and clcache:
activate arrow-dev conda install -c conda-forge ninja pip install git+https://github.com/frerich/clcache.git
After installation is complete, change working directory in
cmd.exe to the root directory of Arrow and
do an out of source build by generating Ninja files:
cd cpp mkdir build cd build cmake -G "Ninja" ^ -DCMAKE_C_COMPILER=clcache ^ -DCMAKE_CXX_COMPILER=clcache ^ -DARROW_BUILD_TESTS=ON ^ -DGTest_SOURCE=BUNDLED .. cmake --build . --config Release
CMAKE_CXX_COMPILER in the command line
cmake is the preferred method of using
clcache. Alternatively, you
CXX environment variables before calling
... set CC=clcache set CXX=clcache cmake -G "Ninja" ^ ...
Building with NMake¶
Change working directory in
cmd.exe to the root directory of Arrow and
do an out of source build using
cd cpp mkdir build cd build cmake -G "NMake Makefiles" .. nmake
Building on MSYS2¶
You can build on MSYS2 terminal,
cmd.exe or PowerShell terminal.
On MSYS2 terminal:
cd cpp mkdir build cd build cmake -G "MSYS Makefiles" .. make
cmd.exe or PowerShell terminal, you can use the following batch
setlocal REM For 64bit set MINGW_PACKAGE_PREFIX=mingw-w64-x86_64 set MINGW_PREFIX=c:\msys64\mingw64 set MSYSTEM=MINGW64 set PATH=%MINGW_PREFIX%\bin;c:\msys64\usr\bin;%PATH% rmdir /S /Q cpp\build mkdir cpp\build pushd cpp\build cmake -G "MSYS Makefiles" .. || exit /B make || exit /B popd
To build a Debug version of Arrow, you should have pre-installed a Debug
version of Boost. It’s recommended to configure
cmake with the following
variables for Debug build:
-DARROW_BOOST_USE_SHARED=OFF: enables static linking with boost debug libs and simplifies run-time loading of 3rd parties
-DBOOST_ROOT: sets the root directory of boost libs. (Optional)
-DBOOST_LIBRARYDIR: sets the directory with boost lib files. (Optional)
The command line to build Arrow in Debug mode will look something like this:
cd cpp mkdir build cd build cmake .. -G "Visual Studio 14 2015" -A x64 ^ -DARROW_BOOST_USE_SHARED=OFF ^ -DCMAKE_BUILD_TYPE=Debug ^ -DBOOST_ROOT=C:/local/boost_1_63_0 ^ -DBOOST_LIBRARYDIR=C:/local/boost_1_63_0/lib64-msvc-14.0 cmake --build . --config Debug
Windows dependency resolution issues¶
Because Windows uses
.lib files for both static and dynamic linking of
dependencies, the static library sometimes may be named something different
%PACKAGE%_static.lib to distinguish itself. If you are statically
linking some dependencies, we provide some options
To get the latest build instructions, you can reference ci/appveyor-built.bat, which is used by automated Appveyor builds.
Replicating Appveyor Builds¶
For people more familiar with linux development but need to replicate a failing
appveyor build, here are some rough notes from replicating the
Static_Crt_Build (make unittest will probably still fail but many unit
tests can be made with there individual make targets).
Microsoft offers trial VMs for Windows with Microsoft Visual Studio. Download and install a version.
Run the VM and install Git, CMake, and Miniconda or Anaconda (these instructions assume Anaconda). Also install the “Build Tools for Visual Studio”. Make sure to select the C++ toolchain in the installer wizard, and reboot after installation.
Download pre-built Boost debug binaries and install it.
Run this from an Anaconda/Miniconda command prompt (not PowerShell prompt), and make sure to run “vcvarsall.bat x64” first. The location of vcvarsall.bat will depend, it may be under a different path than commonly indicated, e.g. “
C:\Program Files (x86)\Microsoft Visual Studio\2019\BuildTools\VC\Auxiliary\Build\vcvarsall.bat” with the 2019 build tools.
cd $EXTRACT_BOOST_DIRECTORY .\bootstrap.bat @rem This is for static libraries needed for static_crt_build in appveyor .\b2 link=static --with-filesystem --with-regex --with-system install @rem this should put libraries and headers in c:\Boost
@rem this might differ for miniconda C:\Users\User\Anaconda3\Scripts\activate
Clone and change directories to the arrow source code (you might need to install git).
Setup environment variables:
@rem Change the build type based on which appveyor job you want. SET JOB=Static_Crt_Build SET GENERATOR=Ninja SET APPVEYOR_BUILD_WORKER_IMAGE=Visual Studio 2017 SET USE_CLCACHE=false SET ARROW_BUILD_GANDIVA=OFF SET ARROW_LLVM_VERSION=8.0.* SET PYTHON=3.6 SET ARCH=64 SET PATH=C:\Users\User\Anaconda3;C:\Users\User\Anaconda3\Scripts;C:\Users\User\Anaconda3\Library\bin;%PATH% SET BOOST_LIBRARYDIR=C:\Boost\lib SET BOOST_ROOT=C:\Boost
Run appveyor scripts:
conda install -c conda-forge --file .\ci\conda_env_cpp.yml .\ci\appveyor-cpp-setup.bat @rem this might fail but at this point most unit tests should be buildable by there individual targets @rem see next line for example. .\ci\appveyor-cpp-build.bat @rem you can also just invoke cmake directly with the desired options cmake --build . --config Release --target arrow-compute-hash-test