A FileFormat holds information about how to read and parse the files
included in a Dataset. There are subclasses corresponding to the supported
file formats (ParquetFileFormat and IpcFileFormat).
FileFormat$create() takes the following arguments:
format: A string identifier of the file format. Currently supported values:
"parquet"
"ipc"/"arrow"/"feather", all aliases for each other; for Feather, note that only version 2 files are supported
"csv"/"text", aliases for the same thing (because comma is the default delimiter for text files
"tsv", equivalent to passing format = "text", delimiter = "\t"
...: Additional format-specific options
`format = "parquet"``:
dict_columns: Names of columns which should be read as dictionaries.
Any Parquet options from FragmentScanOptions.
format = "text": see CsvParseOptions. Note that you can specify them either
with the Arrow C++ library naming ("delimiter", "quoting", etc.) or the
readr-style naming used in read_csv_arrow() ("delim", "quote", etc.).
Not all readr options are currently supported; please file an issue if
you encounter one that arrow should support. Also, the following options are
supported. From CsvReadOptions:
skip_rows
column_names
autogenerate_column_names
From CsvFragmentScanOptions (these values can be overridden at scan time):
convert_options: a CsvConvertOptions
block_size
It returns the appropriate subclass of FileFormat (e.g. ParquetFileFormat)
## Semi-colon delimited files # Set up directory for examples tf <- tempfile() dir.create(tf) on.exit(unlink(tf)) write.table(mtcars, file.path(tf, "file1.txt"), sep = ";", row.names = FALSE) # Create FileFormat object format <- FileFormat$create(format = "text", delimiter = ";") open_dataset(tf, format = format) #> FileSystemDataset with 1 csv file #> mpg: double #> cyl: int64 #> disp: double #> hp: int64 #> drat: double #> wt: double #> qsec: double #> vs: int64 #> am: int64 #> gear: int64 #> carb: int64