5 Defining Data Types

5.1 Introduction

As discussed in previous chapters, Arrow automatically infers the most appropriate data type when reading in data or converting R objects to Arrow objects. However, you might want to manually tell Arrow which data types to use, for example, to ensure interoperability with databases and data warehouse systems. This chapter includes recipes for:

• changing the data types of existing Arrow objects
• defining data types during the process of creating Arrow objects

A table showing the default mappings between R and Arrow data types can be found in R data type to Arrow data type mappings.

A table containing Arrow data types, and their R equivalents can be found in Arrow data type to R data type mapping.

5.2 Update data type of an existing Arrow Array

You want to change the data type of an existing Arrow Array.

5.2.1 Solution

# Create an Array to cast
integer_arr <- Array$create(1:5)

# Cast to an unsigned int8 type
uint_arr <- integer_arr$cast(target_type = uint8())

uint_arr

## Array
## <uint8>
## [
##   1,
##   2,
##   3,
##   4,
##   5
## ]

5.2.2 Discussion

There are some data types which are not compatible with each other. Errors will occur if you try to cast between incompatible data types.

int_arr <- Array$create(1:5)
int_arr$cast(target_type = binary())

## Error: NotImplemented: Unsupported cast from int32 to binary using function cast_binary

5.3 Update data type of a field in an existing Arrow Table

You want to change the type of one or more fields in an existing Arrow Table.

5.3.1 Solution

# Set up a tibble to use in this example
oscars <- tibble::tibble(
  actor = c("Katharine Hepburn", "Meryl Streep", "Jack Nicholson"),
  num_awards = c(4, 3, 3)
)

# Convert tibble to an Arrow table
oscars_arrow <- arrow_table(oscars)

# The default mapping from numeric column "num_awards" is to a double
oscars_arrow

## Table
## 3 rows x 2 columns
## $actor <string>
## $num_awards <double>

# Set up schema with "num_awards" as integer
oscars_schema <- schema(actor = string(), num_awards = int16())

# Cast to an int16
oscars_arrow_int <- oscars_arrow$cast(target_schema = oscars_schema)

oscars_arrow_int

## Table
## 3 rows x 2 columns
## $actor <string>
## $num_awards <int16>

5.3.2 Discussion

There are some Arrow data types which do not have any R equivalent. Attempting to cast to these data types or using a schema which contains them will result in an error.

# Set up a tibble to use in this example
oscars <- tibble::tibble(
  actor = c("Katharine Hepburn", "Meryl Streep", "Jack Nicholson"),
  num_awards = c(4, 3, 3)
)

# Convert tibble to an Arrow table
oscars_arrow <- arrow_table(oscars)

# Set up schema with "num_awards" as float16 which doesn't have an R equivalent
oscars_schema_invalid <- schema(actor = string(), num_awards = float16())

# The default mapping from numeric column "num_awards" is to a double
oscars_arrow$cast(target_schema = oscars_schema_invalid)

## Error: NotImplemented: Unsupported cast from double to halffloat using function cast_half_float

5.4 Specify data types when creating an Arrow table from an R object

You want to manually specify Arrow data types when converting an object from a data frame to an Arrow object.

5.4.1 Solution

# Set up a tibble to use in this example
oscars <- tibble::tibble(
  actor = c("Katharine Hepburn", "Meryl Streep", "Jack Nicholson"),
  num_awards = c(4, 3, 3)
)

# Set up schema with "num_awards" as integer
oscars_schema <- schema(actor = string(), num_awards = int16())

# create arrow Table containing data and schema
oscars_data_arrow <- arrow_table(oscars, schema = oscars_schema)

oscars_data_arrow

## Table
## 3 rows x 2 columns
## $actor <string>
## $num_awards <int16>

5.5 Specify data types when reading in files

You want to manually specify Arrow data types when reading in files.

5.5.1 Solution

# Set up a tibble to use in this example
oscars <- tibble::tibble(
  actor = c("Katharine Hepburn", "Meryl Streep", "Jack Nicholson"),
  num_awards = c(4, 3, 3)
)

# write dataset to disk
write_dataset(oscars, path = "oscars_data")

# Set up schema with "num_awards" as integer
oscars_schema <- schema(actor = string(), num_awards = int16())

# read the dataset in, using the schema instead of inferring the type automatically
oscars_dataset_arrow <- open_dataset("oscars_data", schema = oscars_schema)

oscars_dataset_arrow

## FileSystemDataset with 1 Parquet file
## actor: string
## num_awards: int16