A FileFormat holds information about how to read and parse the files included in a Dataset. There are subclasses corresponding to the supported file formats (ParquetFileFormat and IpcFileFormat).


FileFormat$create() takes the following arguments:

  • format: A string identifier of the file format. Currently supported values:

    • "parquet"

    • "ipc"/"arrow"/"feather", all aliases for each other; for Feather, note that only version 2 files are supported

    • "csv"/"text", aliases for the same thing (because comma is the default delimiter for text files

    • "tsv", equivalent to passing format = "text", delimiter = "\t"

  • ...: Additional format-specific options

    `format = "parquet"``:

    • dict_columns: Names of columns which should be read as dictionaries.

    • Any Parquet options from FragmentScanOptions.

    format = "text": see CsvParseOptions. Note that you can specify them either with the Arrow C++ library naming ("delimiter", "quoting", etc.) or the readr-style naming used in read_csv_arrow() ("delim", "quote", etc.). Not all readr options are currently supported; please file an issue if you encounter one that arrow should support. Also, the following options are supported. From CsvReadOptions:

It returns the appropriate subclass of FileFormat (e.g. ParquetFileFormat)


## Semi-colon delimited files # Set up directory for examples tf <- tempfile() dir.create(tf) on.exit(unlink(tf)) write.table(mtcars, file.path(tf, "file1.txt"), sep = ";", row.names = FALSE) # Create FileFormat object format <- FileFormat$create(format = "text", delimiter = ";") open_dataset(tf, format = format)
#> FileSystemDataset with 1 csv file #> mpg: double #> cyl: int64 #> disp: double #> hp: int64 #> drat: double #> wt: double #> qsec: double #> vs: int64 #> am: int64 #> gear: int64 #> carb: int64