On macOS and Windows, when you install.packages("arrow"), you get a binary package that contains Arrow’s C++ dependencies along with it. On Linux, install.packages() retrieves a source package that has to be compiled locally, and C++ dependencies need to be resolved as well. Generally for R packages with C++ dependencies, this requires either installing system packages, which you may not have privileges to do, or building the C++ dependencies separately, which introduces all sorts of additional ways for things to go wrong.
Our goal is to make install.packages("arrow") “just work” for as many Linux distributions, versions, and configurations as possible. This document describes how it works and the options for fine-tuning Linux installation. The intended audience for this document is arrow R package users on Linux, not developers. If you’re contributing to the Arrow project, see `vignette(“developing”, package = “arrow”) for guidance on setting up your development environment.
Note also that if you use conda to manage your R environment, this document does not apply. You can conda install -c conda-forge --strict-channel-priority r-arrow and you’ll get the latest official release of the R package along with any C++ dependencies.
Having trouble installing
arrow? See the “Troubleshooting” section below.
Install the latest release of arrow from CRAN with
install.packages("arrow")Daily development builds, which are not official releases, can be installed from the Ursa Labs repository:
install.packages("arrow", repos = "https://arrow-r-nightly.s3.amazonaws.com")or for conda users via:
conda install -c arrow-nightlies -c conda-forge --strict-channel-priority r-arrowYou can also install the R package from a git checkout:
git clone https://github.com/apache/arrow
cd arrow/r
R CMD INSTALL .If you don’t already have the Arrow C++ libraries on your system, when installing the R package from source, it will also download and build the Arrow C++ libraries for you. To speed installation up, you can set
export LIBARROW_BINARY=trueto look for C++ binaries prebuilt for your Linux distribution/version. Alternatively, you can set
export LIBARROW_MINIMAL=falseto build the Arrow libraries from source with optional features such as compression libraries enabled. This will increase the build time but provides many useful features. Prebuilt binaries are built with this flag enabled, so you get the full functionality by using them as well.
Both of these variables are also set this way if you have the NOT_CRAN=true environment variable set.
If you already have arrow installed and want to upgrade to a different version, install a development build, or try to reinstall and fix issues with Linux C++ binaries, you can call install_arrow(). install_arrow() provides some convenience wrappers around the various environment variables described below. This function is part of the arrow package, and it is also available as a standalone script, so you can access it for convenience without first installing the package:
source("https://raw.githubusercontent.com/apache/arrow/master/r/R/install-arrow.R")install_arrow() will install from CRAN, while install_arrow(nightly = TRUE) will give you a development build. install_arrow() does not require environment variables to be set in order to satisfy C++ dependencies.
Note that, unlike packages like
tensorflow,blogdown, and others that require external dependencies, you do not need to runinstall_arrow()after a successfularrowinstallation.
The install-arrow.R file also includes the create_package_with_all_dependencies() function. Normally, when installing on a computer with internet access, the build process will download third-party dependencies as needed. This function provides a way to download them in advance. Doing so may be useful when installing Arrow on a computer without internet access. Note that Arrow can be installed on a computer without internet access without doing this, but many useful features will be disabled, as they depend on third-party components. More precisely, arrow::arrow_info()$capabilities() will be FALSE for every capability. One approach to add more capabilities in an offline install is to prepare a package with pre-downloaded dependencies. The create_package_with_all_dependencies() function does this preparation.
If you’re using binary packages you shouldn’t need to follow these steps. You should download the appropriate binary from your package repository, transfer that to the offline computer, and install that. Any OS can create the source bundle, but it cannot be installed on Windows. (Instead, use a standard Windows binary package.)
Note if you’re using RStudio Package Manager on Linux: If you still want to make a source bundle with this function, make sure to set the first repo in options("repos") to be a mirror that contains source packages (that is: something other than the RSPM binary mirror URLs).
arrow package or run source("https://raw.githubusercontent.com/apache/arrow/master/r/R/install-arrow.R")
create_package_with_all_dependencies("my_arrow_pkg.tar.gz")
my_arrow_pkg.tar.gz to the computer without internet accessarrow package from the copied file
install.packages("my_arrow_pkg.tar.gz", dependencies = c("Depends", "Imports", "LinkingTo"))cmake must be availablearrow_info() to check installed capabilitiescpp/thirdparty/download_dependencies.sh may be helpful)ARROW_THIRDPARTY_DEPENDENCY_DIR on the offline computer, pointing to the copied directory.arrow package as usual.The arrow package allows you to work with data in AWS S3 or in other cloud storage system that emulate S3. However, support for working with S3 is not enabled in the default build, and it has additional system requirements. To enable it, set the environment variable LIBARROW_MINIMAL=false or NOT_CRAN=true to choose the full-featured build, or more selectively set ARROW_S3=ON. You also need the following system dependencies:
gcc >= 4.9 or clang >= 3.3; note that the default compiler on CentOS 7 is gcc 4.8.5, which is not sufficientlibcurl-devel (rpm) or libcurl4-openssl-dev (deb)openssl-devel (rpm) or libssl-dev (deb)The prebuilt C++ binaries come with S3 support enabled, so you will need to meet these system requirements in order to use them–the package will not install without them. If you’re building everything from source, the install script will check for the presence of these dependencies and turn off S3 support in the build if the prerequisites are not met–installation will succeed but without S3 functionality. If afterwards you install the missing system requirements, you’ll need to reinstall the package in order to enable S3 support.
In order for the arrow R package to work, it needs the Arrow C++ library. There are a number of ways you can get it: a system package; a library you’ve built yourself outside of the context of installing the R package; or, if you don’t already have it, the R package will attempt to resolve it automatically when it installs.
If you are authorized to install system packages and you’re installing a CRAN release, you may want to use the official Apache Arrow release packages corresponding to the R package version (though there are some drawbacks: see “Troubleshooting” below). See the Arrow project installation page to find pre-compiled binary packages for some common Linux distributions, including Debian, Ubuntu, and CentOS. You’ll need to install libparquet-dev on Debian and Ubuntu, or parquet-devel on CentOS. This will also automatically install the Arrow C++ library as a dependency.
When you install the arrow R package on Linux, it will first attempt to find the Arrow C++ libraries on your system using the pkg-config command. This will find either installed system packages or libraries you’ve built yourself. In order for install.packages("arrow") to work with these system packages, you’ll need to install them before installing the R package.
If no Arrow C++ libraries are found on the system, the R package installation script will next attempt to download prebuilt static Arrow C++ libraries that match your both your local operating system and arrow R package version. C++ binaries will only be retrieved if you have set the environment variable LIBARROW_BINARY or NOT_CRAN. If found, they will be downloaded and bundled when your R package compiles. For a list of supported distributions and versions, see the arrow-r-nightly project.
If no C++ library binary is found, it will attempt to build it locally. First, it will also look to see if you are in a checkout of the apache/arrow git repository and thus have the C++ source there. Otherwise, it builds from the C++ files included in the package. Depending on your system, building Arrow C++ from source may be slow.
For the specific mechanics of how all this works, see the R package configure script, which calls tools/nixlibs.R.
If the C++ library is built from source, inst/build_arrow_static.sh is executed. This build script is also what is used to generate the prebuilt binaries.
This subsection contains information which is likely to be most relevant mostly to Arrow developers and is not necessary for Arrow users to install Arrow.
There are a number of scripts that are triggered when R CMD INSTALL . is run. For Arrow users, these should all just work without configuration and pull in the most complete pieces (e.g. official binaries that we host).
An overview of these scripts is shown below:
configure and configure.win - these scripts are triggered during R CMD INSTALL . on non-Windows and Windows platforms, respectively. They handle finding the Arrow library, setting up the build variables necessary, and writing the package Makevars file that is used to compile the C++ code in the R package.
tools/nixlibs.R - this script is sometimes called by configure on Linux (or on any non-windows OS with the environment variable FORCE_BUNDLED_BUILD=true). This sets up the build process for our bundled builds (which is the default on linux). The operative logic is at the end of the script, but it will do the following (and it will stop with the first one that succeeds and some of the steps are only checked if they are enabled via an environment variable):
arrow/r/libarrow-{version}, use that to link against if it exists.*** Proceed without C++ dependencies (this is an error and the package will not work, but if you see this message you know the previous steps have not succeeded/were not enabled)inst/build_arrow_static.sh - called by tools/nixlibs.R when the Arrow library is being built. It builds Arrow for a bundled, static build, and mirrors the steps described in the “Arrow R Developer Guide” vignette
The intent is that install.packages("arrow") will just work and handle all C++ dependencies, but depending on your system, you may have better results if you tune one of several parameters. Here are some known complications and ways to address them.
If you see a message like
------------------------- NOTE ---------------------------
There was an issue preparing the Arrow C++ libraries.
See https://arrow.apache.org/docs/r/articles/install.html
---------------------------------------------------------in the output when the package fails to install, that means that installation failed to retrieve or build C++ libraries compatible with the current version of the R package.
It is expected that C++ dependencies should be built successfully on all Linux distributions, so you should not see this message. If you do, please check the “Known installation issues” below to see if any apply. If none apply, set the environment variable ARROW_R_DEV=TRUE so that details on what failed are shown, and try installing again. Then, please report an issue and include the full verbose installation output.
If a system library or other installed Arrow is found but it doesn’t match the R package version (for example, you have libarrow 1.0.0 on your system and are installing R package 2.0.0), it is likely that the R bindings will fail to compile. Because the Apache Arrow project is under active development, is it essential that versions of the C++ and R libraries match. When install.packages("arrow") has to download the C++ libraries, the install script ensures that you fetch the C++ libraries that correspond to your R package version. However, if you are using Arrow libraries already on your system, version match isn’t guaranteed.
To fix version mismatch, you can either update your system packages to match the R package version, or set the environment variable ARROW_USE_PKG_CONFIG=FALSE to tell the configure script not to look for system Arrow packages. (The latter is the default of install_arrow().) System packages are available corresponding to all CRAN releases but not for nightly or dev versions, so depending on the R package version you’re installing, system packages may not be an option.
Note also that once you have a working R package installation based on system (shared) libraries, if you update your system Arrow, you’ll need to reinstall the R package to match its version. Similarly, if you’re using Arrow system libraries, running update.packages() after a new release of the arrow package will likely fail unless you first update the system packages.
If the R package finds and downloads a prebuilt binary of the C++ library, but then the arrow package can’t be loaded, perhaps with “undefined symbols” errors, please report an issue. This is likely a compiler mismatch and may be resolvable by setting some environment variables to instruct R to compile the packages to match the C++ library.
A workaround would be to set the environment variable LIBARROW_BINARY=FALSE and retry installation: this value instructs the package to build the C++ library from source instead of downloading the prebuilt binary. That should guarantee that the compiler settings match.
If a prebuilt binary wasn’t found for your operating system but you think it should have been, check the logs for a message that says *** Unable to identify current OS/version, or a message that says *** No C++ binaries found for an invalid OS. If you see either, please report an issue. You may also set the environment variable ARROW_R_DEV=TRUE for additional debug messages.
A workaround would be to set the environment variable LIBARROW_BINARY to a distribution-version that exists in the Ursa Labs repository. Setting LIBARROW_BINARY is also an option when there’s not an exact match for your OS but a similar version would work, such as if you’re on ubuntu-18.10 and there’s only a binary for ubuntu-18.04.
If that workaround works for you, and you believe that it should work for everyone else too, you may propose adding an entry to this lookup table. This table is checked during the installation process and tells the script to use binaries built on a different operating system/version because they’re known to work.
If building the C++ library from source fails, check the error message. (If you don’t see an error message, only the ----- NOTE -----, set the environment variable ARROW_R_DEV=TRUE to increase verbosity and retry installation.) The install script should work everywhere, so if the C++ library fails to compile, please report an issue so that we can improve the script.
On CentOS, if you are using a more modern devtoolset, you may need to set the environment variables CC and CXX either in the shell or in R’s Makeconf. For CentOS 7 and above, both the Arrow system packages and the C++ binaries for R are built with the default system compilers. If you want to use either of these and you have a devtoolset installed, set CC=/usr/bin/gcc CXX=/usr/bin/g++ to use the system compilers instead of the devtoolset. Alternatively, if you want to build arrow with the newer devtoolset compilers, set both ARROW_USE_PKG_CONFIG and LIBARROW_BINARY to false so that you build the Arrow C++ from source using those compilers. Compiler mismatch between the arrow system libraries and the R package may cause R to segfault when arrow package functions are used. See discussions here and here.
If you have multiple versions of zstd installed on your system, installation by building the C++ from source may fail with an undefined symbols error. Workarounds include (1) setting LIBARROW_BINARY to use a C++ binary; (2) setting ARROW_WITH_ZSTD=OFF to build without zstd; or (3) uninstalling the conflicting zstd. See discussion here.
Some features are optional when you build Arrow from source. With the exception of ARROW_S3, these are all ON by default in the bundled C++ build, but you can set them to OFF to disable them.
ARROW_S3: If set to ON S3 support will be built as long as the dependencies are met; if they are not met, the build script will turn this OFF
ARROW_JEMALLOC for the jemalloc memory allocatorARROW_MIMALLOC for the mimalloc memmory allocatorARROW_PARQUETARROW_DATASETARROW_JSON for the JSON parsing libraryARROW_WITH_RE2 for the RE2 regular expression library, used in some string compute functionsARROW_WITH_UTF8PROC for the UTF8Proc string library, used in many other string compute functionsARROW_JSON for JSON parsingARROW_WITH_BROTLI, ARROW_WITH_BZ2, ARROW_WITH_LZ4, ARROW_WITH_SNAPPY, ARROW_WITH_ZLIB, and ARROW_WITH_ZSTD for various compression algorithmsThere are a number of other variables that affect the configure script and the bundled build script. By default, these are all unset. All boolean variables are case-insensitive.
ARROW_USE_PKG_CONFIG: If set to false, the configure script won’t look for Arrow libraries on your system and instead will look to download/build them. Use this if you have a version mismatch between installed system libraries and the version of the R package you’re installing.LIBARROW_BINARY: If set to true, the script will try to download a binary C++ library built for your operating system. You may also set it to some other string, a related “distro-version” that has binaries built that work for your OS. If no binary is found, installation will fall back to building C++ dependencies from source.LIBARROW_BUILD: If set to false, the build script will not attempt to build the C++ from source. This means you will only get a working arrow R package if a prebuilt binary is found. Use this if you want to avoid compiling the C++ library, which may be slow and resource-intensive, and ensure that you only use a prebuilt binary.LIBARROW_MINIMAL: If set to false, the build script will enable some optional features, including compression libraries, S3 support, and additional alternative memory allocators. This will increase the source build time but results in a more fully functional library.NOT_CRAN: If this variable is set to true, as the devtools package does, the build script will set LIBARROW_BINARY=true and LIBARROW_MINIMAL=false unless those environment variables are already set. This provides for a more complete and fast installation experience for users who already have NOT_CRAN=true as part of their workflow, without requiring additional environment variables to be set.ARROW_R_DEV: If set to true, more verbose messaging will be printed in the build script. arrow::install_arrow(verbose = TRUE) sets this. This variable also is needed if you’re modifying C++ code in the package: see the developer guide vignette.LIBARROW_DEBUG_DIR: If the C++ library building from source fails (cmake), there may be messages telling you to check some log file in the build directory. However, when the library is built during R package installation, that location is in a temp directory that is already deleted. To capture those logs, set this variable to an absolute (not relative) path and the log files will be copied there. The directory will be created if it does not exist.CMAKE: When building the C++ library from source, you can specify a /path/to/cmake to use a different version than whatever is found on the $PATH
As mentioned above, please report an issue if you encounter ways to improve this. If you find that your Linux distribution or version is not supported, we welcome the contribution of Docker images (hosted on Docker Hub) that we can use in our continuous integration. These Docker images should be minimal, containing only R and the dependencies it requires. (For reference, see the images that R-hub uses.)
You can test the arrow R package installation using the docker-compose setup included in the apache/arrow git repository. For example,
R_ORG=rhub R_IMAGE=ubuntu-gcc-release R_TAG=latest docker-compose build r
R_ORG=rhub R_IMAGE=ubuntu-gcc-release R_TAG=latest docker-compose run rinstalls the arrow R package, including the C++ source build, on the rhub/ubuntu-gcc-release image.