# arrow 5.0.0 2021-07-29

## More dplyr

• There are now more than 250 compute functions available for use in dplyr::filter(), mutate(), etc. Additions in this release include:

• String operations: strsplit() and str_split(); strptime(); paste(), paste0(), and str_c(); substr() and str_sub(); str_like(); str_pad(); stri_reverse()
• Date/time operations: lubridate methods such as year(), month(), wday(), and so on
• Math: logarithms (log() et al.); trigonometry (sin(), cos(), et al.); abs(); sign(); pmin() and pmax(); ceiling(), floor(), and trunc()
• Conditional functions, with some limitations on input type in this release: ifelse() and if_else() for all but Decimal types; case_when() for logical, numeric, and temporal types only; coalesce() for all but lists/structs. Note also that in this release, factors/dictionaries are converted to strings in these functions.
• is.* functions are supported and can be used inside relocate()
• The print method for arrow_dplyr_query now includes the expression and the resulting type of columns derived by mutate().
• transmute() now errors if passed arguments .keep, .before, or .after, for consistency with the behavior of dplyr on data.frames.

## CSV writing

• write_csv_arrow() to use Arrow to write a data.frame to a single CSV file
• write_dataset(format = "csv", ...) to write a Dataset to CSVs, including with partitioning

• RecordBatch columns can now be added, replaced, or removed by assigning (<-) with either $ or [[ • Similarly, Schema can now be edited by assigning in new types. This enables using the CSV reader to detect the schema of a file, modify the Schema object for any columns that you want to read in as a different type, and then use that Schema to read the data. • Better validation when creating a Table with a schema, with columns of different lengths, and with scalar value recycling • Reading Parquet files in Japanese or other multi-byte locales on Windows no longer hangs (workaround for a bug in libstdc++; thanks @yutannihilation for the persistence in discovering this!) • If you attempt to read string data that has embedded nul (\0) characters, the error message now informs you that you can set options(arrow.skip_nul = TRUE) to strip them out. It is not recommended to set this option by default since this code path is significantly slower, and most string data does not contain nuls. • read_json_arrow() now accepts a schema: read_json_arrow("file.json", schema = schema(col_a = float64(), col_b = string())) ## Installation and configuration • The R package can now support working with an Arrow C++ library that has additional features (such as dataset, parquet, string libraries) disabled, and the bundled build script enables setting environment variables to disable them. See vignette("install", package = "arrow") for details. This allows a faster, smaller package build in cases where that is useful, and it enables a minimal, functioning R package build on Solaris. • On macOS, it is now possible to use the same bundled C++ build that is used by default on Linux, along with all of its customization parameters, by setting the environment variable FORCE_BUNDLED_BUILD=true. • arrow now uses the mimalloc memory allocator by default on macOS, if available (as it is in CRAN binaries), instead of jemalloc. There are configuration issues with jemalloc on macOS, and benchmark analysis shows that this has negative effects on performance, especially on memory-intensive workflows. jemalloc remains the default on Linux; mimalloc is default on Windows. • Setting the ARROW_DEFAULT_MEMORY_POOL environment variable to switch memory allocators now works correctly when the Arrow C++ library has been statically linked (as is usually the case when installing from CRAN). • The arrow_info() function now reports on the additional optional features, as well as the detected SIMD level. If key features or compression libraries are not enabled in the build, arrow_info() will refer to the installation vignette for guidance on how to install a more complete build, if desired. • If you attempt to read a file that was compressed with a codec that your Arrow build does not contain support for, the error message now will tell you how to reinstall Arrow with that feature enabled. • A new vignette about developer environment setup vignette("developing", package = "arrow"). • When building from source, you can use the environment variable ARROW_HOME to point to a specific directory where the Arrow libraries are. This is similar to passing INCLUDE_DIR and LIB_DIR. # arrow 3.0.0 2021-01-27 ## Python and Flight • Flight methods flight_get() and flight_put() (renamed from push_data() in this release) can handle both Tables and RecordBatches • flight_put() gains an overwrite argument to optionally check for the existence of a resource with the the same name • list_flights() and flight_path_exists() enable you to see available resources on a Flight server • Schema objects now have r_to_py and py_to_r methods • Schema metadata is correctly preserved when converting Tables to/from Python ## Enhancements • Arithmetic operations (+, *, etc.) are supported on Arrays and ChunkedArrays and can be used in filter expressions in Arrow dplyr pipelines • Table columns can now be added, replaced, or removed by assigning (<-) with either $ or [[
• Column names of Tables and RecordBatches can be renamed by assigning names()
• Large string types can now be written to Parquet files
• The pronouns .data and .env are now fully supported in Arrow dplyr pipelines.
• Option arrow.skip_nul (default FALSE, as in base::scan()) allows conversion of Arrow string (utf8()) type data containing embedded nul \0 characters to R. If set to TRUE, nuls will be stripped and a warning is emitted if any are found.
• arrow_info() for an overview of various run-time and build-time Arrow configurations, useful for debugging
• Set environment variable ARROW_DEFAULT_MEMORY_POOL before loading the Arrow package to change memory allocators. Windows packages are built with mimalloc; most others are built with both jemalloc (used by default) and mimalloc. These alternative memory allocators are generally much faster than the system memory allocator, so they are used by default when available, but sometimes it is useful to turn them off for debugging purposes. To disable them, set ARROW_DEFAULT_MEMORY_POOL=system.
• List columns that have attributes on each element are now also included with the metadata that is saved when creating Arrow tables. This allows sf tibbles to faithfully preserved and roundtripped (ARROW-10386).
• R metadata that exceeds 100Kb is now compressed before being written to a table; see schema() for more details.

## Bug fixes

• Fixed a performance regression in converting Arrow string types to R that was present in the 2.0.0 release
• C++ functions now trigger garbage collection when needed
• write_parquet() can now write RecordBatches
• Reading a Table from a RecordBatchStreamReader containing 0 batches no longer crashes
• readr’s problems attribute is removed when converting to Arrow RecordBatch and table to prevent large amounts of metadata from accumulating inadvertently (ARROW-10624)
• Fixed reading of compressed Feather files written with Arrow 0.17 (ARROW-10850)
• SubTreeFileSystem gains a useful print method and no longer errors when printing

## Packaging and installation

• Nightly development versions of the conda r-arrow package are available with conda install -c arrow-nightlies -c conda-forge --strict-channel-priority r-arrow
• Linux installation now safely supports older cmake versions
• Compiler version checking for enabling S3 support correctly identifies the active compiler
• Updated guidance and troubleshooting in vignette("install", package = "arrow"), especially for known CentOS issues
• Operating system detection on Linux uses the distro package. If your OS isn’t correctly identified, please report an issue there.

# arrow 2.0.0 2020-10-20

## Datasets

• write_dataset() to Feather or Parquet files with partitioning. See the end of vignette("dataset", package = "arrow") for discussion and examples.
• Datasets now have head(), tail(), and take ([) methods. head() is optimized but the others may not be performant.
• collect() gains an as_data_frame argument, default TRUE but when FALSE allows you to evaluate the accumulated select and filter query but keep the result in Arrow, not an R data.frame
• read_csv_arrow() supports specifying column types, both with a Schema and with the compact string representation for types used in the readr package. It also has gained a timestamp_parsers argument that lets you express a set of strptime parse strings that will be tried to convert columns designated as Timestamp type.

## AWS S3 support

• S3 support is now enabled in binary macOS and Windows (Rtools40 only, i.e. R >= 4.0) packages. To enable it on Linux, you need the additional system dependencies libcurl and openssl, as well as a sufficiently modern compiler. See vignette("install", package = "arrow") for details.

## Compression

• write_parquet() now supports compression
• codec_is_available() returns TRUE or FALSE whether the Arrow C++ library was built with support for a given compression library (e.g. gzip, lz4, snappy)
• Windows builds now include support for zstd and lz4 compression (#5814, @gnguy)

## Other fixes and improvements

• Arrow null type is now supported
• Factor types are now preserved in round trip through Parquet format (#6135, @yutannihilation)
• Reading an Arrow dictionary type coerces dictionary values to character (as R factor levels are required to be) instead of raising an error
• Many improvements to Parquet function documentation (@karldw, @khughitt)

# arrow 0.15.1 2019-11-04

• This patch release includes bugfixes in the C++ library around dictionary types and Parquet reading.

# arrow 0.15.0 2019-10-07

## Breaking changes

• The R6 classes that wrap the C++ classes are now documented and exported and have been renamed to be more R-friendly. Users of the high-level R interface in this package are not affected. Those who want to interact with the Arrow C++ API more directly should work with these objects and methods. As part of this change, many functions that instantiated these R6 objects have been removed in favor of Class$create() methods. Notably, arrow::array() and arrow::table() have been removed in favor of Array$create() and Table\$create(), eliminating the package startup message about masking base functions. For more information, see the new vignette("arrow").
• Due to a subtle change in the Arrow message format, data written by the 0.15 version libraries may not be readable by older versions. If you need to send data to a process that uses an older version of Arrow (for example, an Apache Spark server that hasn’t yet updated to Arrow 0.15), you can set the environment variable ARROW_PRE_0_15_IPC_FORMAT=1.
• The as_tibble argument in the read_*() functions has been renamed to as_data_frame (ARROW-6337, @jameslamb)
• The arrow::Column class has been removed, as it was removed from the C++ library

## New features

• Table and RecordBatch objects have S3 methods that enable you to work with them more like data.frames. Extract columns, subset, and so on. See ?Table and ?RecordBatch for examples.
• Initial implementation of bindings for the C++ File System API. (ARROW-6348)
• Compressed streams are now supported on Windows (ARROW-6360), and you can also specify a compression level (ARROW-6533)

• Parquet file reading is much, much faster, thanks to improvements in the Arrow C++ library.
• read_csv_arrow() supports more parsing options, including col_names, na, quoted_na, and skip
• read_parquet() and read_feather() can ingest data from a raw vector (ARROW-6278)
• File readers now properly handle paths that need expanding, such as ~/file.parquet (ARROW-6323)
• Improved support for creating types in a schema: the types’ printed names (e.g. “double”) are guaranteed to be valid to use in instantiating a schema (e.g. double()), and time types can be created with human-friendly resolution strings (“ms”, “s”, etc.). (ARROW-6338, ARROW-6364)

# arrow 0.14.1 2019-08-05

Initial CRAN release of the arrow package. Key features include:

• Read and write support for various file formats, including Parquet, Feather/Arrow, CSV, and JSON.
• API bindings to the C++ library for Arrow data types and objects, as well as mapping between Arrow types and R data types.
• Tools for helping with C++ library configuration and installation.