Extension arrays are wrappers around regular Arrow Array objects that provide some customized behaviour and/or storage. A common use-case for extension types is to define a customized conversion between an an Arrow Array and an R object when the default conversion is slow or looses metadata important to the interpretation of values in the array. For most types, the built-in vctrs extension type is probably sufficient.
new_extension_type(
storage_type,
extension_name,
extension_metadata = raw(),
type_class = ExtensionType
)
new_extension_array(storage_array, extension_type)
register_extension_type(extension_type)
reregister_extension_type(extension_type)
unregister_extension_type(extension_name)
The data type of the underlying storage array.
The extension name. This should be namespaced using "dot" syntax (i.e., "some_package.some_type"). The namespace "arrow" is reserved for extension types defined by the Apache Arrow libraries.
A raw()
or character()
vector containing the
serialized version of the type. Chatacter vectors must be length 1 and
are converted to UTF-8 before converting to raw()
.
An R6::R6Class whose $new()
class method will be
used to construct a new instance of the type.
An Array object of the underlying storage.
An ExtensionType instance.
new_extension_type()
returns an ExtensionType instance according
to the type_class
specified.
new_extension_array()
returns an ExtensionArray whose $type
corresponds to extension_type
.
register_extension_type()
, unregister_extension_type()
and reregister_extension_type()
return NULL
, invisibly.
These functions create, register, and unregister ExtensionType and ExtensionArray objects. To use an extension type you will have to:
Define an R6::R6Class that inherits from ExtensionType and reimplement
one or more methods (e.g., deserialize_instance()
).
Make a type constructor function (e.g., my_extension_type()
) that calls
new_extension_type()
to create an R6 instance that can be used as a
data type elsewhere in the package.
Make an array constructor function (e.g., my_extension_array()
) that
calls new_extension_array()
to create an Array instance of your
extension type.
Register a dummy instance of your extension type created using
you constructor function using register_extension_type()
.
If defining an extension type in an R package, you will probably want to
use reregister_extension_type()
in that package's .onLoad()
hook
since your package will probably get reloaded in the same R session
during its development and register_extension_type()
will error if
called twice for the same extension_name
. For an example of an
extension type that uses most of these features, see
vctrs_extension_type()
.
# Create the R6 type whose methods control how Array objects are
# converted to R objects, how equality between types is computed,
# and how types are printed.
QuantizedType <- R6::R6Class(
"QuantizedType",
inherit = ExtensionType,
public = list(
# methods to access the custom metadata fields
center = function() private$.center,
scale = function() private$.scale,
# called when an Array of this type is converted to an R vector
as_vector = function(extension_array) {
if (inherits(extension_array, "ExtensionArray")) {
unquantized_arrow <-
(extension_array$storage()$cast(float64()) / private$.scale) +
private$.center
as.vector(unquantized_arrow)
} else {
super$as_vector(extension_array)
}
},
# populate the custom metadata fields from the serialized metadata
deserialize_instance = function() {
vals <- as.numeric(strsplit(self$extension_metadata_utf8(), ";")[[1]])
private$.center <- vals[1]
private$.scale <- vals[2]
}
),
private = list(
.center = NULL,
.scale = NULL
)
)
# Create a helper type constructor that calls new_extension_type()
quantized <- function(center = 0, scale = 1, storage_type = int32()) {
new_extension_type(
storage_type = storage_type,
extension_name = "arrow.example.quantized",
extension_metadata = paste(center, scale, sep = ";"),
type_class = QuantizedType
)
}
# Create a helper array constructor that calls new_extension_array()
quantized_array <- function(x, center = 0, scale = 1,
storage_type = int32()) {
type <- quantized(center, scale, storage_type)
new_extension_array(
Array$create((x - center) * scale, type = storage_type),
type
)
}
# Register the extension type so that Arrow knows what to do when
# it encounters this extension type
reregister_extension_type(quantized())
# Create Array objects and use them!
(vals <- runif(5, min = 19, max = 21))
#> [1] 19.50434 20.36954 20.83199 19.05251 19.31069
(array <- quantized_array(
vals,
center = 20,
scale = 2 ^ 15 - 1,
storage_type = int16())
)
#> ExtensionArray
#> <QuantizedType <20;32767>>
#> [
#> -16241,
#> 12108,
#> 27261,
#> -31046,
#> -22586
#> ]
array$type$center()
#> [1] 20
array$type$scale()
#> [1] 32767
as.vector(array)
#> [1] 19.50435 20.36952 20.83197 19.05252 19.31071