Compute Functions#

Datum class#

class Datum#

Variant type for various Arrow C++ data structures.

Public Types

enum Kind#

The kind of datum stored.

Values:

enumerator NONE#

enumerator SCALAR#

enumerator ARRAY#

enumerator CHUNKED_ARRAY#

enumerator RECORD_BATCH#

enumerator TABLE#

Public Functions

Datum() = default#: Empty datum, to be populated elsewhere.

inline Datum(std::shared_ptr<Scalar> value)#: Construct from a Scalar.

inline Datum(std::shared_ptr<ArrayData> value)#: Construct from an ArrayData.

inline Datum(ArrayData arg)#: Construct from an ArrayData.

Datum(const Array &value)#: Construct from an Array.

Datum(const std::shared_ptr<Array> &value)#: Construct from an Array.

Datum(std::shared_ptr<ChunkedArray> value)#: Construct from a ChunkedArray.

Datum(std::shared_ptr<RecordBatch> value)#: Construct from a RecordBatch.

Datum(std::shared_ptr<Table> value)#: Construct from a Table.

explicit Datum(const ChunkedArray &value)#

Construct from a ChunkedArray.

This can be expensive, prefer the shared_ptr<ChunkedArray> constructor

explicit Datum(const RecordBatch &value)#

Construct from a RecordBatch.

This can be expensive, prefer the shared_ptr<RecordBatch> constructor

explicit Datum(const Table &value)#

Construct from a Table.

This can be expensive, prefer the shared_ptr<Table> constructor

template<typename T, bool IsArray = std::is_base_of_v<Array, T>, bool IsScalar = std::is_base_of_v<Scalar, T>, typename = enable_if_t<IsArray || IsScalar>> inline Datum(std::shared_ptr<T> value)#: Cast from concrete subtypes of Array or Scalar to Datum.

template<typename T, typename TV = typename std::remove_reference_t<T>, bool IsArray = std::is_base_of_v<Array, T>, bool IsScalar = std::is_base_of_v<Scalar, T>, typename = enable_if_t<IsArray || IsScalar>> inline Datum(T &&value)#: Cast from concrete subtypes of Array or Scalar to Datum.

template<typename T, typename = enable_if_t<std::is_base_of_v<Scalar, T>>> inline Datum(const T &value)#

Copy from concrete subtypes of Scalar.

The concrete scalar type must be copyable (not all of them are).

explicit Datum(bool value)#: Convenience constructor storing a bool scalar.

explicit Datum(int8_t value)#: Convenience constructor storing an int8 scalar.

explicit Datum(uint8_t value)#: Convenience constructor storing a uint8 scalar.

explicit Datum(int16_t value)#: Convenience constructor storing an int16 scalar.

explicit Datum(uint16_t value)#: Convenience constructor storing a uint16 scalar.

explicit Datum(int32_t value)#: Convenience constructor storing an int32 scalar.

explicit Datum(uint32_t value)#: Convenience constructor storing a uint32 scalar.

explicit Datum(int64_t value)#: Convenience constructor storing an int64 scalar.

explicit Datum(uint64_t value)#: Convenience constructor storing a uint64 scalar.

explicit Datum(float value)#: Convenience constructor storing a float scalar.

explicit Datum(double value)#: Convenience constructor storing a double scalar.

explicit Datum(std::string value)#: Convenience constructor storing a string scalar.

explicit Datum(const char *value)#: Convenience constructor storing a string scalar.

template<template<typename, typename> class StdDuration, typename Rep, typename Period, typename = decltype(DurationScalar{StdDuration<Rep, Period>{}})> inline explicit Datum(StdDuration<Rep, Period> d)#: Convenience constructor for a DurationScalar from std::chrono::duration.

inline Datum::Kind kind() const#: The kind of data stored in Datum.

inline const std::shared_ptr<ArrayData> &array() const#

Retrieve the stored array as ArrayData.

Use make_array() if an Array is desired (which is more expensive).

Throws:: std::bad_variant_access – if the datum is not an array

int64_t TotalBufferSize() const#

The sum of bytes in each buffer referenced by the datum Note: Scalars report a size of 0.

Abstract Function classes#

void PrintTo(const FunctionOptions&, std::ostream*)#

struct Arity#

#include <arrow/compute/function.h>

Contains the number of required arguments for the function.

Naming conventions taken from https://en.wikipedia.org/wiki/Arity.

Public Members

int num_args#: The number of required arguments (or the minimum number for varargs functions).

bool is_varargs = false#: If true, then the num_args is the minimum number of required arguments.

Public Static Functions

static inline Arity Nullary()#: A function taking no arguments.

static inline Arity Unary()#: A function taking 1 argument.

static inline Arity Binary()#: A function taking 2 arguments.

static inline Arity Ternary()#: A function taking 3 arguments.

static inline Arity VarArgs(int min_args = 0)#

A function taking a variable number of arguments.

Parameters:: min_args – [in] the minimum number of arguments required when invoking the function

struct FunctionDoc#

#include <arrow/compute/function.h>

Public Members

std::string summary#

A one-line summary of the function, using a verb.

For example, “Add two numeric arrays or scalars”.

std::string description#: A detailed description of the function, meant to follow the summary.

std::vector<std::string> arg_names#

Symbolic names (identifiers) for the function arguments.

Some bindings may use this to generate nicer function signatures.

std::string options_class#: Name of the options class, if any.

bool options_required#

Whether options are required for function execution.

If false, then either the function does not have an options class or there is a usable default options value.

class FunctionExecutor#

#include <arrow/compute/function.h>

An executor of a function with a preconfigured kernel.

Public Functions

virtual Status Init(const FunctionOptions *options = NULLPTR, ExecContext *exec_ctx = NULLPTR) = 0#

Initialize or re-initialize the preconfigured kernel.

This method may be called zero or more times. Depending on how the FunctionExecutor was obtained, it may already have been initialized.

virtual Result<Datum> Execute(const std::vector<Datum> &args, int64_t length = -1) = 0#

Execute the preconfigured kernel with arguments that must fit it.

The method requires the arguments be castable to the preconfigured types.

Parameters:

args – [in] Arguments to execute the function on
length – [in] Length of arguments batch or -1 to default it. If the function has no parameters, this determines the batch length, defaulting to 0. Otherwise, if the function is scalar, this must equal the argument batch’s inferred length or be -1 to default to it. This is ignored for vector functions.

class Function#

#include <arrow/compute/function.h>

Base class for compute functions.

Function implementations contain a collection of “kernels” which are implementations of the function for specific argument types. Selecting a viable kernel for executing a function is referred to as “dispatching”.

Subclassed by arrow::compute::detail::FunctionImpl< HashAggregateKernel >, arrow::compute::detail::FunctionImpl< ScalarAggregateKernel >, arrow::compute::detail::FunctionImpl< ScalarKernel >, arrow::compute::detail::FunctionImpl< VectorKernel >, arrow::compute::MetaFunction, arrow::compute::detail::FunctionImpl< KernelType >

Public Types

enum Kind#

The kind of function, which indicates in what contexts it is valid for use.

Values:

enumerator SCALAR#

A function that performs scalar data operations on whole arrays of data.

Can generally process Array or Scalar values. The size of the output will be the same as the size (or broadcasted size, in the case of mixing Array and Scalar inputs) of the input.

enumerator VECTOR#: A function with array input and output whose behavior depends on the values of the entire arrays passed, rather than the value of each scalar value.

enumerator SCALAR_AGGREGATE#: A function that computes scalar summary statistics from array input.

enumerator HASH_AGGREGATE#: A function that computes grouped summary statistics from array input and an array of group identifiers.

enumerator META#: A function that dispatches to other functions and does not contain its own kernels.

Public Functions

inline const std::string &name() const#: The name of the kernel. The registry enforces uniqueness of names.

inline Function::Kind kind() const#: The kind of kernel, which indicates in what contexts it is valid for use.

inline const Arity &arity() const#: Contains the number of arguments the function requires, or if the function accepts variable numbers of arguments.

inline const FunctionDoc &doc() const#: Return the function documentation.

virtual int num_kernels() const = 0#: Returns the number of registered kernels for this function.

virtual Result<const Kernel*> DispatchExact(const std::vector<TypeHolder> &types) const#

Return a kernel that can execute the function given the exact argument types (without implicit type casts).

NB: This function is overridden in CastFunction.

virtual Result<const Kernel*> DispatchBest(std::vector<TypeHolder> *values) const#

Return a best-match kernel that can execute the function given the argument types, after implicit casts are applied.

Parameters:: values – [inout] Argument types. An element may be modified to indicate that the returned kernel only approximately matches the input value descriptors; callers are responsible for casting inputs to the type required by the kernel.

virtual Result<std::shared_ptr<FunctionExecutor>> GetBestExecutor(std::vector<TypeHolder> inputs) const#

Get a function executor with a best-matching kernel.

The returned executor will by default work with the default FunctionOptions and KernelContext. If you want to change that, call FunctionExecutor::Init.

virtual Result<Datum> Execute(const std::vector<Datum> &args, const FunctionOptions *options, ExecContext *ctx) const#

Execute the function eagerly with the passed input arguments with kernel dispatch, batch iteration, and memory allocation details taken care of.

If the options pointer is null, then default_options() will be used.

This function can be overridden in subclasses.

inline const FunctionOptions *default_options() const#

Returns the default options for this function.

Whatever option semantics a Function has, implementations must guarantee that default_options() is valid to pass to Execute as options.

inline virtual bool is_pure() const#

Returns the pure property for this function.

Impure functions are those that may return different results for the same input arguments. For example, a function that returns a random number is not pure. An expression containing only pure functions can be simplified by pre-evaluating any sub-expressions that have constant arguments.

class ScalarFunction : public arrow::compute::detail::FunctionImpl<ScalarKernel>#

#include <arrow/compute/function.h>

A function that executes elementwise operations on arrays or scalars, and therefore whose results generally do not depend on the order of the values in the arguments.

Accepts and returns arrays that are all of the same size. These functions roughly correspond to the functions used in SQL expressions.

Public Functions

Status AddKernel(std::vector<InputType> in_types, OutputType out_type, ArrayKernelExec exec, KernelInit init = NULLPTR)#: Add a kernel with given input/output types, no required state initialization, preallocation for fixed-width types, and default null handling (intersect validity bitmaps of inputs).

Status AddKernel(ScalarKernel kernel)#

Add a kernel (function implementation).

Returns error if the kernel’s signature does not match the function’s arity.

inline virtual bool is_pure() const override#: Returns the pure property for this function.

class VectorFunction : public arrow::compute::detail::FunctionImpl<VectorKernel>#

#include <arrow/compute/function.h>

A function that executes general array operations that may yield outputs of different sizes or have results that depend on the whole array contents.

These functions roughly correspond to the functions found in non-SQL array languages like APL and its derivatives.

Public Functions

Status AddKernel(std::vector<InputType> in_types, OutputType out_type, ArrayKernelExec exec, KernelInit init = NULLPTR)#: Add a simple kernel with given input/output types, no required state initialization, no data preallocation, and no preallocation of the validity bitmap.

Status AddKernel(VectorKernel kernel)#

Add a kernel (function implementation).

Returns error if the kernel’s signature does not match the function’s arity.

class ScalarAggregateFunction : public arrow::compute::detail::FunctionImpl<ScalarAggregateKernel>#

#include <arrow/compute/function.h>

Public Functions

Status AddKernel(ScalarAggregateKernel kernel)#

Add a kernel (function implementation).

Returns error if the kernel’s signature does not match the function’s arity.

class HashAggregateFunction : public arrow::compute::detail::FunctionImpl<HashAggregateKernel>#

#include <arrow/compute/function.h>

Public Functions

Status AddKernel(HashAggregateKernel kernel)#

Add a kernel (function implementation).

Returns error if the kernel’s signature does not match the function’s arity.

class MetaFunction : public arrow::compute::Function #

#include <arrow/compute/function.h>

A function that dispatches to other functions.

Must implement MetaFunction::ExecuteImpl.

For Array, ChunkedArray, and Scalar Datum kinds, may rely on the execution of concrete Function types, but must handle other Datum kinds on its own.

Public Functions

inline virtual int num_kernels() const override#: Returns the number of registered kernels for this function.

virtual Result<Datum> Execute(const std::vector<Datum> &args, const FunctionOptions *options, ExecContext *ctx) const override#

Execute the function eagerly with the passed input arguments with kernel dispatch, batch iteration, and memory allocation details taken care of.

If the options pointer is null, then default_options() will be used.

This function can be overridden in subclasses.

class FunctionOptionsType#: #include <arrow/compute/function_options.h>

Extension point for defining options outside libarrow (but still within this project).

class FunctionOptions : public arrow::util::EqualityComparable<FunctionOptions>#

#include <arrow/compute/function_options.h>

Base class for specifying options configuring a function’s behavior, such as error handling.

Public Functions

Result<std::shared_ptr<Buffer>> Serialize() const#: Serialize an options struct to a buffer.

Public Static Functions

static Result<std::unique_ptr<FunctionOptions>> Deserialize(const std::string &type_name, const Buffer &buffer)#

Deserialize an options struct from a buffer.

Note: this will only look for type_name in the default FunctionRegistry; to use a custom FunctionRegistry, look up the FunctionOptionsType, then call FunctionOptionsType::Deserialize().

Function execution#

Result<std::shared_ptr<FunctionExecutor>> GetFunctionExecutor(const std::string &func_name, std::vector<TypeHolder> in_types, const FunctionOptions *options = NULLPTR, FunctionRegistry *func_registry = NULLPTR)#

One-shot executor provider for all types of functions.

This function creates and initializes a FunctionExecutor appropriate for the given function name, input types and function options.

Result<std::shared_ptr<FunctionExecutor>> GetFunctionExecutor(const std::string &func_name, const std::vector<Datum> &args, const FunctionOptions *options = NULLPTR, FunctionRegistry *func_registry = NULLPTR)#

One-shot executor provider for all types of functions.

This function creates and initializes a FunctionExecutor appropriate for the given function name, input types (taken from the Datum arguments) and function options.

Function registry#

class FunctionRegistry#

A mutable central function registry for built-in functions as well as user-defined functions.

Functions are implementations of arrow::compute::Function.

Generally, each function contains kernels which are implementations of a function for a specific argument signature. After looking up a function in the registry, one can either execute it eagerly with Function::Execute or use one of the function’s dispatch methods to pick a suitable kernel for lower-level function execution.

Public Functions

Status CanAddFunction(std::shared_ptr<Function> function, bool allow_overwrite = false)#

Check whether a new function can be added to the registry.

Returns:: Status::KeyError if a function with the same name is already registered.

Status AddFunction(std::shared_ptr<Function> function, bool allow_overwrite = false)#

Add a new function to the registry.

Returns:: Status::KeyError if a function with the same name is already registered.

Status CanAddAlias(const std::string &target_name, const std::string &source_name)#

Check whether an alias can be added for the given function name.

Returns:: Status::KeyError if the function with the given name is not registered.

Status AddAlias(const std::string &target_name, const std::string &source_name)#

Add alias for the given function name.

Returns:: Status::KeyError if the function with the given name is not registered.

Status CanAddFunctionOptionsType(const FunctionOptionsType *options_type, bool allow_overwrite = false)#

Check whether a new function options type can be added to the registry.

Returns:: Status::KeyError if a function options type with the same name is already registered.

Status AddFunctionOptionsType(const FunctionOptionsType *options_type, bool allow_overwrite = false)#

Add a new function options type to the registry.

Returns:: Status::KeyError if a function options type with the same name is already registered.

Result<std::shared_ptr<Function>> GetFunction(const std::string &name) const#: Retrieve a function by name from the registry.

std::vector<std::string> GetFunctionNames() const#

Return vector of all entry names in the registry.

Helpful for displaying a manifest of available functions.

Result<const FunctionOptionsType*> GetFunctionOptionsType(const std::string &name) const#: Retrieve a function options type by name from the registry.

int num_functions() const#: The number of currently registered functions.

const Function *cast_function() const#

The cast function object registered in AddFunction.

Helpful for get cast function as needed.

Public Static Functions

static std::unique_ptr<FunctionRegistry> Make()#

Construct a new registry.

Most users only need to use the global registry.

static std::unique_ptr<FunctionRegistry> Make(FunctionRegistry *parent)#

Construct a new nested registry with the given parent.

Most users only need to use the global registry. The returned registry never changes its parent, even when an operation allows overwriting.

FunctionRegistry *arrow::compute::GetFunctionRegistry()#: Return the process-global function registry.

Convenience functions#

Result<Datum> CallFunction(const std::string &func_name, const std::vector<Datum> &args, const FunctionOptions *options, ExecContext *ctx = NULLPTR)#

One-shot invoker for all types of functions.

Does kernel dispatch, argument checking, iteration of ChunkedArray inputs, and wrapping of outputs.

Result<Datum> CallFunction(const std::string &func_name, const std::vector<Datum> &args, ExecContext *ctx = NULLPTR)#

Variant of CallFunction which uses a function’s default options.

NB: Some functions require FunctionOptions be provided.

Result<Datum> CallFunction(const std::string &func_name, const ExecBatch &batch, const FunctionOptions *options, ExecContext *ctx = NULLPTR)#

One-shot invoker for all types of functions.

Does kernel dispatch, argument checking, iteration of ChunkedArray inputs, and wrapping of outputs.

Result<Datum> CallFunction(const std::string &func_name, const ExecBatch &batch, ExecContext *ctx = NULLPTR)#

Variant of CallFunction which uses a function’s default options.

NB: Some functions require FunctionOptions be provided.

Concrete options classes#

enum class RoundMode : int8_t#

Rounding and tie-breaking modes for round compute functions.

Additional details and examples are provided in compute.rst.

Values:

enumerator DOWN#: Round to nearest integer less than or equal in magnitude (aka “floor”)

enumerator UP#: Round to nearest integer greater than or equal in magnitude (aka “ceil”)

enumerator TOWARDS_ZERO#: Get the integral part without fractional digits (aka “trunc”)

enumerator TOWARDS_INFINITY#: Round negative values with DOWN rule and positive values with UP rule (aka “away from zero”)

enumerator HALF_DOWN#: Round ties with DOWN rule (also called “round half towards negative infinity”)

enumerator HALF_UP#: Round ties with UP rule (also called “round half towards positive infinity”)

enumerator HALF_TOWARDS_ZERO#: Round ties with TOWARDS_ZERO rule (also called “round half away from infinity”)

enumerator HALF_TOWARDS_INFINITY#: Round ties with TOWARDS_INFINITY rule (also called “round half away from zero”)

enumerator HALF_TO_EVEN#: Round ties to nearest even integer.

enumerator HALF_TO_ODD#: Round ties to nearest odd integer.

enum class CalendarUnit : int8_t#

Values:

enumerator NANOSECOND#

enumerator MICROSECOND#

enumerator MILLISECOND#

enumerator SECOND#

enumerator MINUTE#

enumerator HOUR#

enumerator DAY#

enumerator WEEK#

enumerator MONTH#

enumerator QUARTER#

enumerator YEAR#

enum CompareOperator#

Values:

enumerator EQUAL#

enumerator NOT_EQUAL#

enumerator GREATER#

enumerator GREATER_EQUAL#

enumerator LESS#

enumerator LESS_EQUAL#

using CumulativeSumOptions = CumulativeOptions#

class ScalarAggregateOptions : public arrow::compute::FunctionOptions #

#include <arrow/compute/api_aggregate.h>

Control general scalar aggregate kernel behavior.

By default, null values are ignored (skip_nulls = true).

Public Functions

explicit ScalarAggregateOptions(bool skip_nulls = true, uint32_t min_count = 1)#

Public Members

bool skip_nulls#

If true (the default), null values are ignored.

Otherwise, if any value is null, emit null.

uint32_t min_count#: If less than this many non-null values are observed, emit null.

Public Static Functions

static inline ScalarAggregateOptions Defaults()#

Public Static Attributes

static constexpr char const kTypeName[] = "ScalarAggregateOptions"#

class CountOptions : public arrow::compute::FunctionOptions #

#include <arrow/compute/api_aggregate.h>

Control count aggregate kernel behavior.

By default, only non-null values are counted.

Public Types

enum CountMode#

Values:

enumerator ONLY_VALID#: Count only non-null values.

enumerator ONLY_NULL#: Count only null values.

enumerator ALL#: Count both non-null and null values.

Public Functions

explicit CountOptions(CountMode mode = CountMode::ONLY_VALID)#

Public Members

CountMode mode#

Public Static Functions

static inline CountOptions Defaults()#

Public Static Attributes

static constexpr char const kTypeName[] = "CountOptions"#

class ModeOptions : public arrow::compute::FunctionOptions #

#include <arrow/compute/api_aggregate.h>

Control Mode kernel behavior.

Returns top-n common values and counts. By default, returns the most common value and count.

Public Functions

explicit ModeOptions(int64_t n = 1, bool skip_nulls = true, uint32_t min_count = 0)#

Public Members

int64_t n = 1#

bool skip_nulls#

If true (the default), null values are ignored.

Otherwise, if any value is null, emit null.

uint32_t min_count#: If less than this many non-null values are observed, emit null.

Public Static Functions

static inline ModeOptions Defaults()#

Public Static Attributes

static constexpr char const kTypeName[] = "ModeOptions"#

class VarianceOptions : public arrow::compute::FunctionOptions #

#include <arrow/compute/api_aggregate.h>

Control Delta Degrees of Freedom (ddof) of Variance and Stddev kernel.

The divisor used in calculations is N - ddof, where N is the number of elements. By default, ddof is zero, and population variance or stddev is returned.

Public Functions

explicit VarianceOptions(int ddof = 0, bool skip_nulls = true, uint32_t min_count = 0)#

Public Members

int ddof = 0#

bool skip_nulls#

If true (the default), null values are ignored.

Otherwise, if any value is null, emit null.

uint32_t min_count#: If less than this many non-null values are observed, emit null.

Public Static Functions

static inline VarianceOptions Defaults()#

Public Static Attributes

static constexpr char const kTypeName[] = "VarianceOptions"#

class SkewOptions : public arrow::compute::FunctionOptions #

#include <arrow/compute/api_aggregate.h>

Control Skew and Kurtosis kernel behavior.

Public Functions

explicit SkewOptions(bool skip_nulls = true, bool biased = true, uint32_t min_count = 0)#

Public Members

bool skip_nulls#

If true (the default), null values are ignored.

Otherwise, if any value is null, emit null.

bool biased#

If true (the default), the calculated value is biased.

If false, the calculated value includes a correction factor to reduce bias, making it more accurate for small sample sizes.

uint32_t min_count#: If less than this many non-null values are observed, emit null.

Public Static Functions

static inline SkewOptions Defaults()#

Public Static Attributes

static constexpr char const kTypeName[] = "SkewOptions"#

class QuantileOptions : public arrow::compute::FunctionOptions #

#include <arrow/compute/api_aggregate.h>

Control Quantile kernel behavior.

By default, returns the median value.

Public Types

enum Interpolation#

Interpolation method to use when quantile lies between two data points.

Values:

enumerator LINEAR#

enumerator LOWER#

enumerator HIGHER#

enumerator NEAREST#

enumerator MIDPOINT#

Public Functions

explicit QuantileOptions(double q = 0.5, enum Interpolation interpolation = LINEAR, bool skip_nulls = true, uint32_t min_count = 0)#

explicit QuantileOptions(std::vector<double> q, enum Interpolation interpolation = LINEAR, bool skip_nulls = true, uint32_t min_count = 0)#

Public Members

std::vector<double> q#: probability level of quantile must be between 0 and 1 inclusive

enum Interpolation interpolation#

bool skip_nulls#

If true (the default), null values are ignored.

Otherwise, if any value is null, emit null.

uint32_t min_count#: If less than this many non-null values are observed, emit null.

Public Static Functions

static inline QuantileOptions Defaults()#

Public Static Attributes

static constexpr char const kTypeName[] = "QuantileOptions"#

class TDigestOptions : public arrow::compute::FunctionOptions #

#include <arrow/compute/api_aggregate.h>

Control TDigest approximate quantile kernel behavior.

By default, returns the median value.

Public Functions

explicit TDigestOptions(double q = 0.5, uint32_t delta = 100, uint32_t buffer_size = 500, bool skip_nulls = true, uint32_t min_count = 0)#

explicit TDigestOptions(std::vector<double> q, uint32_t delta = 100, uint32_t buffer_size = 500, bool skip_nulls = true, uint32_t min_count = 0)#

Public Members

std::vector<double> q#: probability level of quantile must be between 0 and 1 inclusive

uint32_t delta#: compression parameter, default 100

uint32_t buffer_size#: input buffer size, default 500

bool skip_nulls#

If true (the default), null values are ignored.

Otherwise, if any value is null, emit null.

uint32_t min_count#: If less than this many non-null values are observed, emit null.

Public Static Functions

static inline TDigestOptions Defaults()#

Public Static Attributes

static constexpr char const kTypeName[] = "TDigestOptions"#

class PivotWiderOptions : public arrow::compute::FunctionOptions #

#include <arrow/compute/api_aggregate.h>

Control Pivot kernel behavior.

These options apply to the “pivot_wider” and “hash_pivot_wider” functions.

Constraints:

The corresponding Aggregate::target must have two FieldRef elements; the first one points to the pivot key column, the second points to the pivoted data column.
The pivot key column can be string, binary or integer; its values will be matched against key_names in order to dispatch the pivoted data into the output. If the pivot key column is not string-like, the key_names will be cast to the pivot key type.

“pivot_wider” example

Assuming the following two input columns with types utf8 and int16 (respectively):

width   |  11
height  |  13

and the options PivotWiderOptions(.key_names = {"height", "width"})

then the output will be a scalar with the type struct{"height": int16, "width": int16} and the value {"height": 13, "width": 11}.

“hash_pivot_wider” example

Assuming the following input with schema {"group": int32, "key": utf8, "value": int16}:

 group |  key     |  value
-----------------------------
  |  height  |    11
  |  width   |    12
  |  width   |    13
  |  height  |    14
  |  depth   |    15

and the following settings:

a hash grouping key “group”
Aggregate( .function = “hash_pivot_wider”, .options = PivotWiderOptions(.key_names = {“height”, “width”}), .target = {“key”, “value”}, .name = {“properties”})

then the output will have the schema {"group": int32, "properties": struct{"height": int16, "width": int16}} and the following value:

 group |     properties
       |  height  |   width
-----------------------------
  1    |   11     |    12
  2    |   null   |    13
  3    |   14     |    null

Public Types

enum UnexpectedKeyBehavior#

Configure the behavior of pivot keys not in key_names

Values:

enumerator kIgnore#: Unexpected pivot keys are ignored silently.

enumerator kRaise#: Unexpected pivot keys return a KeyError.

Public Functions

explicit PivotWiderOptions(std::vector<std::string> key_names, UnexpectedKeyBehavior unexpected_key_behavior = kIgnore)#

PivotWiderOptions()#

Public Members

std::vector<std::string> key_names#: The values expected in the pivot key column.

UnexpectedKeyBehavior unexpected_key_behavior = kIgnore #: The behavior when pivot keys not in key_names are encountered.

Public Static Functions

static inline PivotWiderOptions Defaults()#

Public Static Attributes

static constexpr char const kTypeName[] = "PivotWiderOptions"#

class IndexOptions : public arrow::compute::FunctionOptions #

#include <arrow/compute/api_aggregate.h>

Control Index kernel behavior.

Public Functions

explicit IndexOptions(std::shared_ptr<Scalar> value)#

IndexOptions()#

Public Members

std::shared_ptr<Scalar> value#

Public Static Attributes

static constexpr char const kTypeName[] = "IndexOptions"#

struct Aggregate#

#include <arrow/compute/api_aggregate.h>

Configure a grouped aggregation.

Public Functions

Aggregate() = default#

inline Aggregate(std::string function, std::shared_ptr<FunctionOptions> options, std::vector<FieldRef> target, std::string name = "")#

inline Aggregate(std::string function, std::shared_ptr<FunctionOptions> options, FieldRef target, std::string name = "")#

inline Aggregate(std::string function, FieldRef target, std::string name)#

inline Aggregate(std::string function, std::string name)#

Public Members

std::string function#: the name of the aggregation function

std::shared_ptr<FunctionOptions> options#: options for the aggregation function

std::vector<FieldRef> target#: zero or more fields to which aggregations will be applied

std::string name#: optional output field name for aggregations

class ArithmeticOptions : public arrow::compute::FunctionOptions #

#include <arrow/compute/api_scalar.h>

Public Functions

explicit ArithmeticOptions(bool check_overflow = false)#

Public Members

bool check_overflow#

Public Static Attributes

static constexpr char const kTypeName[] = "ArithmeticOptions"#

class ElementWiseAggregateOptions : public arrow::compute::FunctionOptions #

#include <arrow/compute/api_scalar.h>

Public Functions

explicit ElementWiseAggregateOptions(bool skip_nulls = true)#

Public Members

bool skip_nulls#

Public Static Functions

static inline ElementWiseAggregateOptions Defaults()#

Public Static Attributes

static constexpr char const kTypeName[] = "ElementWiseAggregateOptions"#

class RoundOptions : public arrow::compute::FunctionOptions #

#include <arrow/compute/api_scalar.h>

Public Functions

explicit RoundOptions(int64_t ndigits = 0, RoundMode round_mode = RoundMode::HALF_TO_EVEN)#

Public Members

int64_t ndigits#: Rounding precision (number of digits to round to)

RoundMode round_mode#: Rounding and tie-breaking mode.

Public Static Functions

static inline RoundOptions Defaults()#

Public Static Attributes

static constexpr char const kTypeName[] = "RoundOptions"#

class RoundBinaryOptions : public arrow::compute::FunctionOptions #

#include <arrow/compute/api_scalar.h>

Public Functions

explicit RoundBinaryOptions(RoundMode round_mode = RoundMode::HALF_TO_EVEN)#

Public Members

RoundMode round_mode#: Rounding and tie-breaking mode.

Public Static Functions

static inline RoundBinaryOptions Defaults()#

Public Static Attributes

static constexpr char const kTypeName[] = "RoundBinaryOptions"#

class RoundTemporalOptions : public arrow::compute::FunctionOptions #

#include <arrow/compute/api_scalar.h>

Public Functions

explicit RoundTemporalOptions(int multiple = 1, CalendarUnit unit = CalendarUnit::DAY, bool week_starts_monday = true, bool ceil_is_strictly_greater = false, bool calendar_based_origin = false)#

Public Members

int multiple#: Number of units to round to.

CalendarUnit unit#: The unit used for rounding of time.

bool week_starts_monday#: What day does the week start with (Monday=true, Sunday=false)

bool ceil_is_strictly_greater#

Enable this flag to return a rounded value that is strictly greater than the input.

For example: ceiling 1970-01-01T00:00:00 to 3 hours would yield 1970-01-01T03:00:00 if set to true and 1970-01-01T00:00:00 if set to false. This applies for ceiling only.

bool calendar_based_origin#

By default time is rounded to a multiple of units since 1970-01-01T00:00:00.

By setting calendar_based_origin to true, time will be rounded to a number of units since the last greater calendar unit. For example: rounding to a multiple of days since the beginning of the month or to hours since the beginning of the day. Exceptions: week and quarter are not used as greater units, therefore days will will be rounded to the beginning of the month not week. Greater unit of week is year. Note that ceiling and rounding might change sorting order of an array near greater unit change. For example rounding YYYY-mm-dd 23:00:00 to 5 hours will ceil and round to YYYY-mm-dd+1 01:00:00 and floor to YYYY-mm-dd 20:00:00. On the other hand YYYY-mm-dd+1 00:00:00 will ceil, round and floor to YYYY-mm-dd+1 00:00:00. This can break the order of an already ordered array.

Public Static Functions

static inline RoundTemporalOptions Defaults()#

Public Static Attributes

static constexpr char const kTypeName[] = "RoundTemporalOptions"#

class RoundToMultipleOptions : public arrow::compute::FunctionOptions #

#include <arrow/compute/api_scalar.h>

Public Functions

explicit RoundToMultipleOptions(double multiple = 1.0, RoundMode round_mode = RoundMode::HALF_TO_EVEN)#

explicit RoundToMultipleOptions(std::shared_ptr<Scalar> multiple, RoundMode round_mode = RoundMode::HALF_TO_EVEN)#

Public Members

std::shared_ptr<Scalar> multiple#

Rounding scale (multiple to round to).

Should be a positive numeric scalar of a type compatible with the argument to be rounded. The cast kernel is used to convert the rounding multiple to match the result type.

RoundMode round_mode#: Rounding and tie-breaking mode.

Public Static Functions

static inline RoundToMultipleOptions Defaults()#

Public Static Attributes

static constexpr char const kTypeName[] = "RoundToMultipleOptions"#

class JoinOptions : public arrow::compute::FunctionOptions #

#include <arrow/compute/api_scalar.h>

Options for var_args_join.

Public Types

enum NullHandlingBehavior#

How to handle null values. (A null separator always results in a null output.)

Values:

enumerator EMIT_NULL#: A null in any input results in a null in the output.

enumerator SKIP#: Nulls in inputs are skipped.

enumerator REPLACE#: Nulls in inputs are replaced with the replacement string.

Public Functions

explicit JoinOptions(NullHandlingBehavior null_handling = EMIT_NULL, std::string null_replacement = "")#

Public Members

NullHandlingBehavior null_handling#

std::string null_replacement#

Public Static Functions

static inline JoinOptions Defaults()#

Public Static Attributes

static constexpr char const kTypeName[] = "JoinOptions"#

class MatchSubstringOptions : public arrow::compute::FunctionOptions #

#include <arrow/compute/api_scalar.h>

Public Functions

explicit MatchSubstringOptions(std::string pattern, bool ignore_case = false)#

MatchSubstringOptions()#

Public Members

std::string pattern#: The exact substring (or regex, depending on kernel) to look for inside input values.

bool ignore_case#: Whether to perform a case-insensitive match.

Public Static Attributes

static constexpr char const kTypeName[] = "MatchSubstringOptions"#

class SplitOptions : public arrow::compute::FunctionOptions #

#include <arrow/compute/api_scalar.h>

Public Functions

explicit SplitOptions(int64_t max_splits = -1, bool reverse = false)#

Public Members

int64_t max_splits#: Maximum number of splits allowed, or unlimited when -1.

bool reverse#: Start splitting from the end of the string (only relevant when max_splits != -1)

Public Static Attributes

static constexpr char const kTypeName[] = "SplitOptions"#

class SplitPatternOptions : public arrow::compute::FunctionOptions #

#include <arrow/compute/api_scalar.h>

Public Functions

explicit SplitPatternOptions(std::string pattern, int64_t max_splits = -1, bool reverse = false)#

SplitPatternOptions()#

Public Members

std::string pattern#: The exact substring to split on.

int64_t max_splits#: Maximum number of splits allowed, or unlimited when -1.

bool reverse#: Start splitting from the end of the string (only relevant when max_splits != -1)

Public Static Attributes

static constexpr char const kTypeName[] = "SplitPatternOptions"#

class ReplaceSliceOptions : public arrow::compute::FunctionOptions #

#include <arrow/compute/api_scalar.h>

Public Functions

explicit ReplaceSliceOptions(int64_t start, int64_t stop, std::string replacement)#

ReplaceSliceOptions()#

Public Members

int64_t start#: Index to start slicing at.

int64_t stop#: Index to stop slicing at.

std::string replacement#: String to replace the slice with.

Public Static Attributes

static constexpr char const kTypeName[] = "ReplaceSliceOptions"#

class ReplaceSubstringOptions : public arrow::compute::FunctionOptions #

#include <arrow/compute/api_scalar.h>

Public Functions

explicit ReplaceSubstringOptions(std::string pattern, std::string replacement, int64_t max_replacements = -1)#

ReplaceSubstringOptions()#

Public Members

std::string pattern#: Pattern to match, literal, or regular expression depending on which kernel is used.

std::string replacement#: String to replace the pattern with.

int64_t max_replacements#: Max number of substrings to replace (-1 means unbounded)

Public Static Attributes

static constexpr char const kTypeName[] = "ReplaceSubstringOptions"#

class ExtractRegexOptions : public arrow::compute::FunctionOptions #

#include <arrow/compute/api_scalar.h>

Public Functions

explicit ExtractRegexOptions(std::string pattern)#

ExtractRegexOptions()#

Public Members

std::string pattern#: Regular expression with named capture fields.

Public Static Attributes

static constexpr char const kTypeName[] = "ExtractRegexOptions"#

class ExtractRegexSpanOptions : public arrow::compute::FunctionOptions #

#include <arrow/compute/api_scalar.h>

Public Functions

explicit ExtractRegexSpanOptions(std::string pattern)#

ExtractRegexSpanOptions()#

Public Members

std::string pattern#: Regular expression with named capture fields.

Public Static Attributes

static constexpr char const kTypeName[] = "ExtractRegexSpanOptions"#

class SetLookupOptions : public arrow::compute::FunctionOptions #

#include <arrow/compute/api_scalar.h>

Options for IsIn and IndexIn functions.

Public Types

enum NullMatchingBehavior#

How to handle null values.

Values:

enumerator MATCH#: MATCH, any null in value_set is successfully matched in the input.

enumerator SKIP#: SKIP, any null in value_set is ignored and nulls in the input produce null (IndexIn) or false (IsIn) values in the output.

enumerator EMIT_NULL#: EMIT_NULL, any null in value_set is ignored and nulls in the input produce null (IndexIn and IsIn) values in the output.

enumerator INCONCLUSIVE#

INCONCLUSIVE, null values are regarded as unknown values, which is sql-compatible.

nulls in the input produce null (IndexIn and IsIn) values in the output. Besides, if value_set contains a null, non-null unmatched values in the input also produce null values (IndexIn and IsIn) in the output.

Public Functions

explicit SetLookupOptions(Datum value_set, NullMatchingBehavior = MATCH)#

SetLookupOptions()#

explicit SetLookupOptions(Datum value_set, bool skip_nulls)#

NullMatchingBehavior GetNullMatchingBehavior() const#

Public Members

Datum value_set#: The set of values to look up input values into.

NullMatchingBehavior null_matching_behavior#

std::optional<bool> skip_nulls#

Whether nulls in value_set count for lookup.

If true, any null in value_set is ignored and nulls in the input produce null (IndexIn) or false (IsIn) values in the output. If false, any null in value_set is successfully matched in the input.

Public Static Attributes

static constexpr char const kTypeName[] = "SetLookupOptions"#

class StructFieldOptions : public arrow::compute::FunctionOptions #

#include <arrow/compute/api_scalar.h>

Options for struct_field function.

Public Functions

explicit StructFieldOptions(std::vector<int> indices)#

explicit StructFieldOptions(std::initializer_list<int>)#

explicit StructFieldOptions(FieldRef field_ref)#

StructFieldOptions()#

Public Members

FieldRef field_ref#: The FieldRef specifying what to extract from struct or union.

Public Static Attributes

static constexpr char const kTypeName[] = "StructFieldOptions"#

class StrptimeOptions : public arrow::compute::FunctionOptions #

#include <arrow/compute/api_scalar.h>

Public Functions

explicit StrptimeOptions(std::string format, TimeUnit::type unit, bool error_is_null = false)#

StrptimeOptions()#

Public Members

std::string format#: The desired format string.

TimeUnit::type unit#: The desired time resolution.

bool error_is_null#: Return null on parsing errors if true or raise if false.

Public Static Attributes

static constexpr char const kTypeName[] = "StrptimeOptions"#

class StrftimeOptions : public arrow::compute::FunctionOptions #

#include <arrow/compute/api_scalar.h>

Public Functions

explicit StrftimeOptions(std::string format, std::string locale = "C")#

StrftimeOptions()#

Public Members

std::string format#: The desired format string.

std::string locale#: The desired output locale string.

Public Static Attributes

static constexpr char const kTypeName[] = "StrftimeOptions"#

static constexpr const char *kDefaultFormat = "%Y-%m-%dT%H:%M:%S"#

class PadOptions : public arrow::compute::FunctionOptions #

#include <arrow/compute/api_scalar.h>

Public Functions

explicit PadOptions(int64_t width, std::string padding = " ", bool lean_left_on_odd_padding = true)#

PadOptions()#

Public Members

int64_t width#: The desired string length.

std::string padding#: What to pad the string with. Should be one codepoint (Unicode)/byte (ASCII).

bool lean_left_on_odd_padding = true#

What to do if there is an odd number of padding characters (in case of centered padding).

Defaults to aligning on the left (i.e. adding the extra padding character on the right)

Public Static Attributes

static constexpr char const kTypeName[] = "PadOptions"#

class TrimOptions : public arrow::compute::FunctionOptions #

#include <arrow/compute/api_scalar.h>

Public Functions

explicit TrimOptions(std::string characters)#

TrimOptions()#

Public Members

std::string characters#: The individual characters to be trimmed from the string.

Public Static Attributes

static constexpr char const kTypeName[] = "TrimOptions"#

class SliceOptions : public arrow::compute::FunctionOptions #

#include <arrow/compute/api_scalar.h>

Public Functions

explicit SliceOptions(int64_t start, int64_t stop = std::numeric_limits<int64_t>::max(), int64_t step = 1)#

SliceOptions()#

Public Members

int64_t start#

int64_t stop#

int64_t step#

Public Static Attributes

static constexpr char const kTypeName[] = "SliceOptions"#

class ListSliceOptions : public arrow::compute::FunctionOptions #

#include <arrow/compute/api_scalar.h>

Public Functions

explicit ListSliceOptions(int64_t start, std::optional<int64_t> stop = std::nullopt, int64_t step = 1, std::optional<bool> return_fixed_size_list = std::nullopt)#

ListSliceOptions()#

Public Members

int64_t start#: The start of list slicing.

std::optional<int64_t> stop#: Optional stop of list slicing. If not set, then slice to end. (NotImplemented)

int64_t step#: Slicing step.

std::optional<bool> return_fixed_size_list#

Public Static Attributes

static constexpr char const kTypeName[] = "ListSliceOptions"#

class NullOptions : public arrow::compute::FunctionOptions #

#include <arrow/compute/api_scalar.h>

Public Functions

explicit NullOptions(bool nan_is_null = false)#

Public Members

bool nan_is_null#

Public Static Functions

static inline NullOptions Defaults()#

Public Static Attributes

static constexpr char const kTypeName[] = "NullOptions"#

struct CompareOptions#

#include <arrow/compute/api_scalar.h>

Public Functions

inline explicit CompareOptions(CompareOperator op)#

inline CompareOptions()#

Public Members

enum CompareOperator op#

class MakeStructOptions : public arrow::compute::FunctionOptions #

#include <arrow/compute/api_scalar.h>

Public Functions

MakeStructOptions(std::vector<std::string> n, std::vector<bool> r, std::vector<std::shared_ptr<const KeyValueMetadata>> m)#

explicit MakeStructOptions(std::vector<std::string> n)#

MakeStructOptions()#

Public Members

std::vector<std::string> field_names#: Names for wrapped columns.

std::vector<bool> field_nullability#: Nullability bits for wrapped columns.

std::vector<std::shared_ptr<const KeyValueMetadata>> field_metadata#: Metadata attached to wrapped columns.

Public Static Attributes

static constexpr char const kTypeName[] = "MakeStructOptions"#

struct DayOfWeekOptions : public arrow::compute::FunctionOptions #

#include <arrow/compute/api_scalar.h>

Public Functions

explicit DayOfWeekOptions(bool count_from_zero = true, uint32_t week_start = 1)#

Public Members

bool count_from_zero#: Number days from 0 if true and from 1 if false.

uint32_t week_start#

What day does the week start with (Monday=1, Sunday=7).

The numbering is unaffected by the count_from_zero parameter.

Public Static Functions

static inline DayOfWeekOptions Defaults()#

Public Static Attributes

static constexpr char const kTypeName[] = "DayOfWeekOptions"#

struct AssumeTimezoneOptions : public arrow::compute::FunctionOptions #

#include <arrow/compute/api_scalar.h>

Used to control timestamp timezone conversion and handling ambiguous/nonexistent times.

Public Types

enum Ambiguous#

How to interpret ambiguous local times that can be interpreted as multiple instants (normally two) due to DST shifts.

AMBIGUOUS_EARLIEST emits the earliest instant amongst possible interpretations. AMBIGUOUS_LATEST emits the latest instant amongst possible interpretations.

Values:

enumerator AMBIGUOUS_RAISE#

enumerator AMBIGUOUS_EARLIEST#

enumerator AMBIGUOUS_LATEST#

enum Nonexistent#

How to handle local times that do not exist due to DST shifts.

NONEXISTENT_EARLIEST emits the instant “just before” the DST shift instant in the given timestamp precision (for example, for a nanoseconds precision timestamp, this is one nanosecond before the DST shift instant). NONEXISTENT_LATEST emits the DST shift instant.

Values:

enumerator NONEXISTENT_RAISE#

enumerator NONEXISTENT_EARLIEST#

enumerator NONEXISTENT_LATEST#

Public Functions

explicit AssumeTimezoneOptions(std::string timezone, Ambiguous ambiguous = AMBIGUOUS_RAISE, Nonexistent nonexistent = NONEXISTENT_RAISE)#

AssumeTimezoneOptions()#

Public Members

std::string timezone#: Timezone to convert timestamps from.

Ambiguous ambiguous#: How to interpret ambiguous local times (due to DST shifts)

Nonexistent nonexistent#: How to interpret nonexistent local times (due to DST shifts)

Public Static Attributes

static constexpr char const kTypeName[] = "AssumeTimezoneOptions"#

struct WeekOptions : public arrow::compute::FunctionOptions #

#include <arrow/compute/api_scalar.h>

Public Functions

explicit WeekOptions(bool week_starts_monday = true, bool count_from_zero = false, bool first_week_is_fully_in_year = false)#

Public Members

bool week_starts_monday#: What day does the week start with (Monday=true, Sunday=false)

bool count_from_zero#: Dates from current year that fall into last ISO week of the previous year return 0 if true and 52 or 53 if false.

bool first_week_is_fully_in_year#: Must the first week be fully in January (true), or is a week that begins on December 29, 30, or 31 considered to be the first week of the new year (false)?

Public Static Functions

static inline WeekOptions Defaults()#

static inline WeekOptions ISODefaults()#

static inline WeekOptions USDefaults()#

Public Static Attributes

static constexpr char const kTypeName[] = "WeekOptions"#

struct Utf8NormalizeOptions : public arrow::compute::FunctionOptions #

#include <arrow/compute/api_scalar.h>

Public Types

enum Form#

Values:

enumerator NFC#

enumerator NFKC#

enumerator NFD#

enumerator NFKD#

Public Functions

explicit Utf8NormalizeOptions(Form form = NFC)#

Public Members

Form form#: The Unicode normalization form to apply.

Public Static Functions

static inline Utf8NormalizeOptions Defaults()#

Public Static Attributes

static constexpr char const kTypeName[] = "Utf8NormalizeOptions"#

class RandomOptions : public arrow::compute::FunctionOptions #

#include <arrow/compute/api_scalar.h>

Public Types

enum Initializer#

Values:

enumerator SystemRandom#

enumerator Seed#

Public Functions

RandomOptions(Initializer initializer, uint64_t seed)#

RandomOptions()#

Public Members

Initializer initializer#: The type of initialization for random number generation - system or provided seed.

uint64_t seed#: The seed value used to initialize the random number generation.

Public Static Functions

static inline RandomOptions FromSystemRandom()#

static inline RandomOptions FromSeed(uint64_t seed)#

static inline RandomOptions Defaults()#

Public Static Attributes

static constexpr char const kTypeName[] = "RandomOptions"#

class MapLookupOptions : public arrow::compute::FunctionOptions #

#include <arrow/compute/api_scalar.h>

Options for map_lookup function.

Public Types

enum Occurrence#

Values:

enumerator FIRST#: Return the first matching value.

enumerator LAST#: Return the last matching value.

enumerator ALL#: Return all matching values.

Public Functions

explicit MapLookupOptions(std::shared_ptr<Scalar> query_key, Occurrence occurrence)#

MapLookupOptions()#

Public Members

std::shared_ptr<Scalar> query_key#: The key to lookup in the map.

Occurrence occurrence#: Whether to return the first, last, or all matching values.

Public Static Attributes

static constexpr char const kTypeName[] = "MapLookupOptions"#

class FilterOptions : public arrow::compute::FunctionOptions #

#include <arrow/compute/api_vector.h>

Public Types

enum NullSelectionBehavior#

Configure the action taken when a slot of the selection mask is null.

Values:

enumerator DROP#: The corresponding filtered value will be removed in the output.

enumerator EMIT_NULL#: The corresponding filtered value will be null in the output.

Public Functions

explicit FilterOptions(NullSelectionBehavior null_selection = DROP)#

Public Members

NullSelectionBehavior null_selection_behavior = DROP #

Public Static Functions

static inline FilterOptions Defaults()#

Public Static Attributes

static constexpr char const kTypeName[] = "FilterOptions"#

class TakeOptions : public arrow::compute::FunctionOptions #

#include <arrow/compute/api_vector.h>

Public Functions

explicit TakeOptions(bool boundscheck = true)#

Public Members

bool boundscheck = true#

Public Static Functions

static inline TakeOptions BoundsCheck()#

static inline TakeOptions NoBoundsCheck()#

static inline TakeOptions Defaults()#

Public Static Attributes

static constexpr char const kTypeName[] = "TakeOptions"#

class DictionaryEncodeOptions : public arrow::compute::FunctionOptions #

#include <arrow/compute/api_vector.h>

Options for the dictionary encode function.

Public Types

enum NullEncodingBehavior#

Configure how null values will be encoded.

Values:

enumerator ENCODE#: The null value will be added to the dictionary with a proper index.

enumerator MASK#: The null value will be masked in the indices array.

Public Functions

explicit DictionaryEncodeOptions(NullEncodingBehavior null_encoding = MASK)#

Public Members

NullEncodingBehavior null_encoding_behavior = MASK #

Public Static Functions

static inline DictionaryEncodeOptions Defaults()#

Public Static Attributes

static constexpr char const kTypeName[] = "DictionaryEncodeOptions"#

class RunEndEncodeOptions : public arrow::compute::FunctionOptions #

#include <arrow/compute/api_vector.h>

Options for the run-end encode function.

Public Functions

explicit RunEndEncodeOptions(std::shared_ptr<DataType> run_end_type = int32())#

Public Members

std::shared_ptr<DataType> run_end_type#

Public Static Functions

static inline RunEndEncodeOptions Defaults()#

Public Static Attributes

static constexpr char const kTypeName[] = "RunEndEncodeOptions"#

class ArraySortOptions : public arrow::compute::FunctionOptions #

#include <arrow/compute/api_vector.h>

Public Functions

explicit ArraySortOptions(SortOrder order = SortOrder::Ascending, NullPlacement null_placement = NullPlacement::AtEnd)#

Public Members

SortOrder order#: Sorting order.

NullPlacement null_placement#: Whether nulls and NaNs are placed at the start or at the end.

Public Static Functions

static inline ArraySortOptions Defaults()#

Public Static Attributes

static constexpr char const kTypeName[] = "ArraySortOptions"#

class SortOptions : public arrow::compute::FunctionOptions #

#include <arrow/compute/api_vector.h>

Public Functions

explicit SortOptions(std::vector<SortKey> sort_keys = {}, NullPlacement null_placement = NullPlacement::AtEnd)#

explicit SortOptions(const Ordering &ordering)#

inline Ordering AsOrdering() &&#

Convenience constructor to create an ordering from SortOptions.

Note: Both classes contain the exact same information. However, sort_options should only be used in a “function options” context while Ordering is used more generally.

inline Ordering AsOrdering() const &#

Public Members

std::vector<SortKey> sort_keys#: Column key(s) to order by and how to order by these sort keys.

NullPlacement null_placement#: Whether nulls and NaNs are placed at the start or at the end.

Public Static Functions

static inline SortOptions Defaults()#

Public Static Attributes

static constexpr char const kTypeName[] = "SortOptions"#

class SelectKOptions : public arrow::compute::FunctionOptions #

#include <arrow/compute/api_vector.h>

SelectK options.

Public Functions

explicit SelectKOptions(int64_t k = -1, std::vector<SortKey> sort_keys = {})#

Public Members

int64_t k#: The number of k elements to keep.

std::vector<SortKey> sort_keys#: Column key(s) to order by and how to order by these sort keys.

Public Static Functions

static inline SelectKOptions Defaults()#

static inline SelectKOptions TopKDefault(int64_t k, std::vector<std::string> key_names = {})#

static inline SelectKOptions BottomKDefault(int64_t k, std::vector<std::string> key_names = {})#

Public Static Attributes

static constexpr char const kTypeName[] = "SelectKOptions"#

class RankOptions : public arrow::compute::FunctionOptions #

#include <arrow/compute/api_vector.h>

Rank options.

Public Types

enum Tiebreaker#

Configure how ties between equal values are handled.

Values:

enumerator Min#: Ties get the smallest possible rank in sorted order.

enumerator Max#: Ties get the largest possible rank in sorted order.

enumerator First#

Ranks are assigned in order of when ties appear in the input.

This ensures the ranks are a stable permutation of the input.

enumerator Dense#: The ranks span a dense [1, M] interval where M is the number of distinct values in the input.

Public Functions

explicit RankOptions(std::vector<SortKey> sort_keys = {}, NullPlacement null_placement = NullPlacement::AtEnd, Tiebreaker tiebreaker = RankOptions::First)#

inline explicit RankOptions(SortOrder order, NullPlacement null_placement = NullPlacement::AtEnd, Tiebreaker tiebreaker = RankOptions::First)#: Convenience constructor for array inputs.

Public Members

std::vector<SortKey> sort_keys#: Column key(s) to order by and how to order by these sort keys.

NullPlacement null_placement#: Whether nulls and NaNs are placed at the start or at the end.

Tiebreaker tiebreaker#: Tiebreaker for dealing with equal values in ranks.

Public Static Functions

static inline RankOptions Defaults()#

Public Static Attributes

static constexpr char const kTypeName[] = "RankOptions"#

class RankQuantileOptions : public arrow::compute::FunctionOptions #

#include <arrow/compute/api_vector.h>

Quantile rank options.

Public Functions

explicit RankQuantileOptions(std::vector<SortKey> sort_keys = {}, NullPlacement null_placement = NullPlacement::AtEnd)#

inline explicit RankQuantileOptions(SortOrder order, NullPlacement null_placement = NullPlacement::AtEnd)#: Convenience constructor for array inputs.

Public Members

std::vector<SortKey> sort_keys#: Column key(s) to order by and how to order by these sort keys.

NullPlacement null_placement#: Whether nulls and NaNs are placed at the start or at the end.

Public Static Functions

static inline RankQuantileOptions Defaults()#

Public Static Attributes

static constexpr char const kTypeName[] = "RankQuantileOptions"#

class PartitionNthOptions : public arrow::compute::FunctionOptions #

#include <arrow/compute/api_vector.h>

Partitioning options for NthToIndices.

Public Functions

explicit PartitionNthOptions(int64_t pivot, NullPlacement null_placement = NullPlacement::AtEnd)#

inline PartitionNthOptions()#

Public Members

int64_t pivot#: The index into the equivalent sorted array of the partition pivot element.

NullPlacement null_placement#: Whether nulls and NaNs are partitioned at the start or at the end.

Public Static Attributes

static constexpr char const kTypeName[] = "PartitionNthOptions"#

class WinsorizeOptions : public arrow::compute::FunctionOptions #

#include <arrow/compute/api_vector.h>

Public Functions

WinsorizeOptions(double lower_limit, double upper_limit)#

inline WinsorizeOptions()#

Public Members

double lower_limit#

The quantile below which all values are replaced with the quantile’s value.

For example, if lower_limit = 0.05, then all values in the lower 5% percentile will be replaced with the 5% percentile value.

double upper_limit#

The quantile above which all values are replaced with the quantile’s value.

For example, if upper_limit = 0.95, then all values in the upper 95% percentile will be replaced with the 95% percentile value.

Public Static Attributes

static constexpr char const kTypeName[] = "WinsorizeOptions"#

class CumulativeOptions : public arrow::compute::FunctionOptions #

#include <arrow/compute/api_vector.h>

Options for cumulative functions.

Note

Also aliased as CumulativeSumOptions for backward compatibility

Public Functions

explicit CumulativeOptions(bool skip_nulls = false)#

explicit CumulativeOptions(double start, bool skip_nulls = false)#

explicit CumulativeOptions(std::shared_ptr<Scalar> start, bool skip_nulls = false)#

Public Members

std::optional<std::shared_ptr<Scalar>> start#

Optional starting value for cumulative operation computation, default depends on the operation and input type.

sum: 0
prod: 1
min: maximum of the input type
max: minimum of the input type
mean: start is ignored because it has no meaning for mean

bool skip_nulls = false#

If true, nulls in the input are ignored and produce a corresponding null output.

When false, the first null encountered is propagated through the remaining output.

Public Static Functions

static inline CumulativeOptions Defaults()#

Public Static Attributes

static constexpr char const kTypeName[] = "CumulativeOptions"#

class PairwiseOptions : public arrow::compute::FunctionOptions #

#include <arrow/compute/api_vector.h>

Options for pairwise functions.

Public Functions

explicit PairwiseOptions(int64_t periods = 1)#

Public Members

int64_t periods = 1#: Periods to shift for applying the binary operation, accepts negative values.

Public Static Functions

static inline PairwiseOptions Defaults()#

Public Static Attributes

static constexpr char const kTypeName[] = "PairwiseOptions"#

class ListFlattenOptions : public arrow::compute::FunctionOptions #

#include <arrow/compute/api_vector.h>

Options for list_flatten function.

Public Functions

explicit ListFlattenOptions(bool recursive = false)#

Public Members

bool recursive = false#: If true, the list is flattened recursively until a non-list array is formed.

Public Static Functions

static inline ListFlattenOptions Defaults()#

Public Static Attributes

static constexpr char const kTypeName[] = "ListFlattenOptions"#

class InversePermutationOptions : public arrow::compute::FunctionOptions #

#include <arrow/compute/api_vector.h>

Options for inverse_permutation function.

Public Functions

explicit InversePermutationOptions(int64_t max_index = -1, std::shared_ptr<DataType> output_type = NULLPTR)#

Public Members

int64_t max_index = -1#

The max value in the input indices to allow.

The length of the function’s output will be this value plus 1. If negative, this value will be set to the length of the input indices minus 1 and the length of the function’s output will be the length of the input indices.

std::shared_ptr<DataType> output_type = NULLPTR#

The type of the output inverse permutation.

If null, the output will be of the same type as the input indices, otherwise must be signed integer type. An invalid error will be reported if this type is not able to store the length of the input indices.

Public Static Functions

static inline InversePermutationOptions Defaults()#

Public Static Attributes

static constexpr char const kTypeName[] = "InversePermutationOptions"#

class ScatterOptions : public arrow::compute::FunctionOptions #

#include <arrow/compute/api_vector.h>

Options for scatter function.

Public Functions

explicit ScatterOptions(int64_t max_index = -1)#

Public Members

int64_t max_index = -1#

The max value in the input indices to allow.

The length of the function’s output will be this value plus 1. If negative, this value will be set to the length of the input indices minus 1 and the length of the function’s output will be the length of the input indices.

Public Static Functions

static inline ScatterOptions Defaults()#

Public Static Attributes

static constexpr char const kTypeName[] = "ScatterOptions"#

class CastOptions : public arrow::compute::FunctionOptions #

#include <arrow/compute/cast.h>

Public Functions

explicit CastOptions(bool safe = true)#

bool is_safe() const#

true if the safety options all match CastOptions::Safe

Note, if this returns false it does not mean is_unsafe will return true

bool is_unsafe() const#

true if the safety options all match CastOptions::Unsafe

Note, if this returns false it does not mean is_safe will return true

Public Members

TypeHolder to_type#

bool allow_int_overflow#

bool allow_time_truncate#

bool allow_time_overflow#

bool allow_decimal_truncate#

bool allow_float_truncate#

bool allow_invalid_utf8#

Public Static Functions

static inline CastOptions Safe(TypeHolder to_type = {})#

static inline CastOptions Unsafe(TypeHolder to_type = {})#

Public Static Attributes

static constexpr char const kTypeName[] = "CastOptions"#

Compute Expressions#

inline bool operator==(const Expression &l, const Expression &r)#

inline bool operator!=(const Expression &l, const Expression &r)#

void PrintTo(const Expression&, std::ostream*)#

Expression literal(Datum lit)#

template<typename Arg> Expression literal(Arg &&arg)#

Expression field_ref(FieldRef ref)#

Expression call(std::string function, std::vector<Expression> arguments, std::shared_ptr<FunctionOptions> options = NULLPTR)#

template<typename Options, typename = typename std::enable_if<std::is_base_of<FunctionOptions, Options>::value>::type> Expression call(std::string function, std::vector<Expression> arguments, Options options)#

std::vector<FieldRef> FieldsInExpression(const Expression&)#: Assemble a list of all fields referenced by an Expression at any depth.

bool ExpressionHasFieldRefs(const Expression&)#: Check if the expression references any fields.

Result<KnownFieldValues> ExtractKnownFieldValues(const Expression &guaranteed_true_predicate)#

Assemble a mapping from field references to known values.

This derives known values from “equal” and “is_null” Expressions referencing a field and a literal.

class Expression#

#include <arrow/compute/expression.h>

An unbound expression which maps a single Datum to another Datum.

An expression is one of

A literal Datum.
A reference to a single (potentially nested) field of the input Datum.
A call to a compute function, with arguments specified by other Expressions.

Public Functions

Result<Expression> Bind(const TypeHolder &in, ExecContext* = NULLPTR) const#

Bind this expression to the given input type, looking up Kernels and field types.

Some expression simplification may be performed and implicit casts will be inserted. Any state necessary for execution will be initialized and returned.

bool IsBound() const#: Return true if all an expression’s field references have explicit types and all of its functions’ kernels are looked up.

bool IsScalarExpression() const#: Return true if this expression is composed only of Scalar literals, field references, and calls to ScalarFunctions.

bool IsNullLiteral() const#: Return true if this expression is literal and entirely null.

bool IsSatisfiable() const#

Return true if this expression could evaluate to true.

Will return true for any unbound or non-boolean Expressions. IsSatisfiable does not (currently) do any canonicalization or simplification of the expression, so even Expressions which are unsatisfiable may spuriously return true here. This function is intended for use in predicate pushdown where a filter expression is simplified by a guarantee, so it assumes that trying to simplify again would be redundant.

const Call *call() const#: Access a Call or return nullptr if this expression is not a call.

const Datum *literal() const#: Access a Datum or return nullptr if this expression is not a literal.

const FieldRef *field_ref() const#: Access a FieldRef or return nullptr if this expression is not a field_ref.

const DataType *type() const#: The type to which this expression will evaluate.

struct Call#: #include <arrow/compute/expression.h>

struct Hash#: #include <arrow/compute/expression.h>

struct Parameter#: #include <arrow/compute/expression.h>

Expression project(std::vector<Expression> values, std::vector<std::string> names)#

Expression equal(Expression lhs, Expression rhs)#

Expression not_equal(Expression lhs, Expression rhs)#

Expression less(Expression lhs, Expression rhs)#

Expression less_equal(Expression lhs, Expression rhs)#

Expression greater(Expression lhs, Expression rhs)#

Expression greater_equal(Expression lhs, Expression rhs)#

Expression is_null(Expression lhs, bool nan_is_null = false)#

Expression is_valid(Expression lhs)#

Expression and_(Expression lhs, Expression rhs)#

Expression and_(const std::vector<Expression>&)#

Expression or_(Expression lhs, Expression rhs)#

Expression or_(const std::vector<Expression>&)#

Expression not_(Expression operand)#

Result<Expression> Canonicalize(Expression, ExecContext* = NULLPTR)#

Weak canonicalization which establishes guarantees for subsequent passes.

Even equivalent Expressions may result in different canonicalized expressions. TODO this could be a strong canonicalization

Result<Expression> FoldConstants(Expression)#

Simplify Expressions based on literal arguments (for example, add(null, x) will always be null so replace the call with a null literal).

Includes early evaluation of all calls whose arguments are entirely literal.

Result<Expression> ReplaceFieldsWithKnownValues(const KnownFieldValues &known_values, Expression)#: Simplify Expressions by replacing with known values of the fields which it references.

Result<Expression> SimplifyWithGuarantee(Expression, const Expression &guaranteed_true_predicate)#

Simplify an expression by replacing subexpressions based on a guarantee: a boolean expression which is guaranteed to evaluate to true.

For example, this is used to remove redundant function calls from a filter expression or to replace a reference to a constant-value field with a literal.

Result<Expression> RemoveNamedRefs(Expression expression)#

Replace all named field refs (e.g.

“x” or “x.y”) with field paths (e.g. [0] or [1,3])

This isn’t usually needed and does not offer any simplification by itself. However, it can be useful to normalize an expression to paths to make it simpler to work with.