Status¶
General¶
[x] SQL Parser
[x] SQL Query Planner
[x] Query Optimizer
[x] Constant folding
[x] Join Reordering
[x] Limit Pushdown
[x] Projection push down
[x] Predicate push down
[x] Type coercion
[x] Parallel query execution
SQL Support¶
[x] Projection
[x] Filter (WHERE)
[x] Filter post-aggregate (HAVING)
[x] Limit
[x] Aggregate
[x] Common math functions
[x] cast
[x] try_cast
[x]
VALUES
listsPostgres compatible String functions
[x] ascii
[x] bit_length
[x] btrim
[x] char_length
[x] character_length
[x] chr
[x] concat
[x] concat_ws
[x] initcap
[x] left
[x] length
[x] lpad
[x] ltrim
[x] octet_length
[x] regexp_replace
[x] repeat
[x] replace
[x] reverse
[x] right
[x] rpad
[x] rtrim
[x] split_part
[x] starts_with
[x] strpos
[x] substr
[x] to_hex
[x] translate
[x] trim
Miscellaneous/Boolean functions
[x] nullif
Approximation functions
[x] approx_distinct
Common date/time functions
[ ] Basic date functions
[ ] Basic time functions
[x] Basic timestamp functions
[x] to_timestamp
nested functions
[x] Array of columns
[x] Schema Queries
[x] SHOW TABLES
[x] SHOW COLUMNS
[x] information_schema.{tables, columns}
[ ] information_schema other views
[x] Sorting
[ ] Nested types
[ ] Lists
[x] Subqueries
[x] Common table expressions
[x] Set Operations
[x] UNION ALL
[x] UNION
[x] INTERSECT
[x] INTERSECT ALL
[x] EXCEPT
[x] EXCEPT ALL
[x] Joins
[x] INNER JOIN
[x] LEFT JOIN
[x] RIGHT JOIN
[x] FULL JOIN
[x] CROSS JOIN
[ ] Window
[x] Empty window
[x] Common window functions
[x] Window with PARTITION BY clause
[x] Window with ORDER BY clause
[ ] Window with FILTER clause
[ ] UDF and UDAF for window functions
Data Sources¶
[x] CSV
[x] Parquet primitive types
[ ] Parquet nested types
Extensibility¶
DataFusion is designed to be extensible at all points. To that end, you can provide your own custom:
[x] User Defined Functions (UDFs)
[x] User Defined Aggregate Functions (UDAFs)
[x] User Defined Table Source (
TableProvider
) for tables[x] User Defined
Optimizer
passes (plan rewrites)[x] User Defined
LogicalPlan
nodes[x] User Defined
ExecutionPlan
nodes
Rust Version Compatbility¶
This crate is tested with the latest stable version of Rust. We do not currently test against other, older versions of the Rust compiler.
Supported SQL¶
This library currently supports many SQL constructs, including
CREATE EXTERNAL TABLE X STORED AS PARQUET LOCATION '...';
to register a table’s locationsSELECT ... FROM ...
together with any expressionALIAS
to name an expressionCAST
to change types, including e.g.Timestamp(Nanosecond, None)
Many mathematical unary and binary expressions such as
+
,/
,sqrt
,tan
,>=
.WHERE
to filterGROUP BY
together with one of the following aggregations:MIN
,MAX
,COUNT
,SUM
,AVG
,CORR
,VAR
,COVAR
,STDDEV
(sample and population)ORDER BY
together with an expression and optionalASC
orDESC
and also optionalNULLS FIRST
orNULLS LAST
Supported Functions¶
DataFusion strives to implement a subset of the PostgreSQL SQL dialect where possible. We explicitly choose a single dialect to maximize interoperability with other tools and allow reuse of the PostgreSQL documents and tutorials as much as possible.
Currently, only a subset of the PostgreSQL dialect is implemented, and we will document any deviations.
Schema Metadata / Information Schema Support¶
DataFusion supports the showing metadata about the tables available. This information can be accessed using the views of the ISO SQL information_schema
schema or the DataFusion specific SHOW TABLES
and SHOW COLUMNS
commands.
More information can be found in the Postgres docs).
To show tables available for use in DataFusion, use the SHOW TABLES
command or the information_schema.tables
view:
> show tables;
+---------------+--------------------+------------+------------+
| table_catalog | table_schema | table_name | table_type |
+---------------+--------------------+------------+------------+
| datafusion | public | t | BASE TABLE |
| datafusion | information_schema | tables | VIEW |
+---------------+--------------------+------------+------------+
> select * from information_schema.tables;
+---------------+--------------------+------------+--------------+
| table_catalog | table_schema | table_name | table_type |
+---------------+--------------------+------------+--------------+
| datafusion | public | t | BASE TABLE |
| datafusion | information_schema | TABLES | SYSTEM TABLE |
+---------------+--------------------+------------+--------------+
To show the schema of a table in DataFusion, use the SHOW COLUMNS
command or the or information_schema.columns
view:
> show columns from t;
+---------------+--------------+------------+-------------+-----------+-------------+
| table_catalog | table_schema | table_name | column_name | data_type | is_nullable |
+---------------+--------------+------------+-------------+-----------+-------------+
| datafusion | public | t | a | Int32 | NO |
| datafusion | public | t | b | Utf8 | NO |
| datafusion | public | t | c | Float32 | NO |
+---------------+--------------+------------+-------------+-----------+-------------+
> select table_name, column_name, ordinal_position, is_nullable, data_type from information_schema.columns;
+------------+-------------+------------------+-------------+-----------+
| table_name | column_name | ordinal_position | is_nullable | data_type |
+------------+-------------+------------------+-------------+-----------+
| t | a | 0 | NO | Int32 |
| t | b | 1 | NO | Utf8 |
| t | c | 2 | NO | Float32 |
+------------+-------------+------------------+-------------+-----------+
Supported Data Types¶
DataFusion uses Arrow, and thus the Arrow type system, for query execution. The SQL types from sqlparser-rs are mapped to Arrow types according to the following table
SQL Data Type |
Arrow DataType |
---|---|
|
|
|
|
|
Not yet supported |
|
Not yet supported |
|
Not yet supported |
|
Not yet supported |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Not yet supported |
|
Not yet supported |
|
Not yet supported |
|
Not yet supported |
|
Not yet supported |
|
Not yet supported |