Arrow Datasets

Arrow C++ provides the concept and implementation of Datasets to work with fragmented data, which can be larger-than-memory, be that due to generating large amounts, reading in from a stream, or having a large file on disk. In this article, you will:

  1. read a multi-file partitioned dataset and put it into a Table,

  2. write out a partitioned dataset from a Table.

Pre-requisites

Before continuing, make sure you have:

  1. An Arrow installation, which you can set up here: Using Arrow C++ in your own project

  2. An understanding of basic Arrow data structures from Basic Arrow Data Structures

To witness the differences, it may be useful to have also read the Arrow File I/O. However, it is not required.

Setup

Before running some computations, we need to fill in a couple gaps:

  1. We need to include necessary headers.

  2. A main() is needed to glue things together.

  3. We need data on disk to play with.

Includes

Before writing C++ code, we need some includes. We’ll get iostream for output, then import Arrow’s compute functionality for each file type we’ll work with in this article:

#include <arrow/api.h>
#include <arrow/dataset/api.h>
// We use Parquet headers for setting up examples; they are not required for using
// datasets.
#include <parquet/arrow/reader.h>
#include <parquet/arrow/writer.h>

#include <iostream>

Main()

For our glue, we’ll use the main() pattern from the previous tutorial on data structures:

int main() {
  arrow::Status st = RunMain();
  if (!st.ok()) {
    std::cerr << st << std::endl;
    return 1;
  }
  return 0;
}

Which, like when we used it before, is paired with a RunMain():

arrow::Status RunMain() {

Generating Files for Reading

We need some files to actually play with. In practice, you’ll likely have some input for your own application. Here, however, we want to explore without the overhead of supplying or finding a dataset, so let’s generate some to make this easy to follow. Feel free to read through this, but the concepts will be visited properly in this article – just copy it in, for now, and realize it ends with a partitioned dataset on disk:

// Generate some data for the rest of this example.
arrow::Result<std::shared_ptr<arrow::Table>> CreateTable() {
  // This code should look familiar from the basic Arrow example, and is not the
  // focus of this example. However, we need data to work on it, and this makes that!
  auto schema =
      arrow::schema({arrow::field("a", arrow::int64()), arrow::field("b", arrow::int64()),
                     arrow::field("c", arrow::int64())});
  std::shared_ptr<arrow::Array> array_a;
  std::shared_ptr<arrow::Array> array_b;
  std::shared_ptr<arrow::Array> array_c;
  arrow::NumericBuilder<arrow::Int64Type> builder;
  ARROW_RETURN_NOT_OK(builder.AppendValues({0, 1, 2, 3, 4, 5, 6, 7, 8, 9}));
  ARROW_RETURN_NOT_OK(builder.Finish(&array_a));
  builder.Reset();
  ARROW_RETURN_NOT_OK(builder.AppendValues({9, 8, 7, 6, 5, 4, 3, 2, 1, 0}));
  ARROW_RETURN_NOT_OK(builder.Finish(&array_b));
  builder.Reset();
  ARROW_RETURN_NOT_OK(builder.AppendValues({1, 2, 1, 2, 1, 2, 1, 2, 1, 2}));
  ARROW_RETURN_NOT_OK(builder.Finish(&array_c));
  return arrow::Table::Make(schema, {array_a, array_b, array_c});
}

// Set up a dataset by writing two Parquet files.
arrow::Result<std::string> CreateExampleParquetDataset(
    const std::shared_ptr<arrow::fs::FileSystem>& filesystem,
    const std::string& root_path) {
  // Much like CreateTable(), this is utility that gets us the dataset we'll be reading
  // from. Don't worry, we also write a dataset in the example proper.
  auto base_path = root_path + "parquet_dataset";
  ARROW_RETURN_NOT_OK(filesystem->CreateDir(base_path));
  // Create an Arrow Table
  ARROW_ASSIGN_OR_RAISE(auto table, CreateTable());
  // Write it into two Parquet files
  ARROW_ASSIGN_OR_RAISE(auto output,
                        filesystem->OpenOutputStream(base_path + "/data1.parquet"));
  ARROW_RETURN_NOT_OK(parquet::arrow::WriteTable(
      *table->Slice(0, 5), arrow::default_memory_pool(), output, 2048));
  ARROW_ASSIGN_OR_RAISE(output,
                        filesystem->OpenOutputStream(base_path + "/data2.parquet"));
  ARROW_RETURN_NOT_OK(parquet::arrow::WriteTable(
      *table->Slice(5), arrow::default_memory_pool(), output, 2048));
  return base_path;
}

arrow::Status PrepareEnv() {
  // Get our environment prepared for reading, by setting up some quick writing.
  ARROW_ASSIGN_OR_RAISE(auto src_table, CreateTable())
  std::shared_ptr<arrow::fs::FileSystem> setup_fs;
  // Note this operates in the directory the executable is built in.
  char setup_path[256];
  char* result = getcwd(setup_path, 256);
  if (result == NULL) {
    return arrow::Status::IOError("Fetching PWD failed.");
  }

  ARROW_ASSIGN_OR_RAISE(setup_fs, arrow::fs::FileSystemFromUriOrPath(setup_path));
  ARROW_ASSIGN_OR_RAISE(auto dset_path, CreateExampleParquetDataset(setup_fs, ""));

  return arrow::Status::OK();
}

In order to actually have these files, make sure the first thing called in RunMain() is our helper function PrepareEnv(), which will get a dataset on disk for us to play with:

  ARROW_RETURN_NOT_OK(PrepareEnv());

Reading a Partitioned Dataset

Reading a Dataset is a distinct task from reading a single file. The task takes more work than reading a single file, due to needing to be able to parse multiple files and/or folders. This process can be broken up into the following steps:

  1. Getting a fs::FileSystem object for the local FS

  2. Create a fs::FileSelector and use it to prepare a dataset::FileSystemDatasetFactory

  3. Build a dataset::Dataset using the dataset::FileSystemDatasetFactory

  4. Use a dataset::Scanner to read into a Table

Preparing a FileSystem Object

In order to begin, we’ll need to be able to interact with the local filesystem. In order to do that, we’ll need an fs::FileSystem object. A fs::FileSystem is an abstraction that lets us use the same interface regardless of using Amazon S3, Google Cloud Storage, or local disk – and we’ll be using local disk. So, let’s declare it:

  // First, we need a filesystem object, which lets us interact with our local
  // filesystem starting at a given path. For the sake of simplicity, that'll be
  // the current directory.
  std::shared_ptr<arrow::fs::FileSystem> fs;

For this example, we’ll have our FileSystem’s base path exist in the same directory as the executable. fs::FileSystemFromUriOrPath() lets us get a fs::FileSystem object for any of the types of supported filesystems. Here, though, we’ll just pass our path:

  // Get the CWD, use it to make the FileSystem object.
  char init_path[256];
  char* result = getcwd(init_path, 256);
  if (result == NULL) {
    return arrow::Status::IOError("Fetching PWD failed.");
  }
  ARROW_ASSIGN_OR_RAISE(fs, arrow::fs::FileSystemFromUriOrPath(init_path));

See also

fs::FileSystem for the other supported filesystems.

Creating a FileSystemDatasetFactory

A fs::FileSystem stores a lot of metadata, but we need to be able to traverse it and parse that metadata. In Arrow, we use a FileSelector to do so:

  // A file selector lets us actually traverse a multi-file dataset.
  arrow::fs::FileSelector selector;

This fs::FileSelector isn’t able to do anything yet. In order to use it, we need to configure it – we’ll have it start any selection in “parquet_dataset,” which is where the environment preparation process has left us a dataset, and set recursive to true, which allows for traversal of folders.

  selector.base_dir = "parquet_dataset";
  // Recursive is a safe bet if you don't know the nesting of your dataset.
  selector.recursive = true;

To get a dataset::Dataset from a fs::FileSystem, we need to prepare a dataset::FileSystemDatasetFactory. This is a long but descriptive name – it’ll make us a factory to get data from our fs::FileSystem. First, we configure it by filling a dataset::FileSystemFactoryOptions struct:

  // Making an options object lets us configure our dataset reading.
  arrow::dataset::FileSystemFactoryOptions options;
  // We'll use Hive-style partitioning. We'll let Arrow Datasets infer the partition
  // schema. We won't set any other options, defaults are fine.
  options.partitioning = arrow::dataset::HivePartitioning::MakeFactory();

There are many file formats, and we have to pick one that will be expected when actually reading. Parquet is what we have on disk, so of course we’ll ask for that when reading:

  auto read_format = std::make_shared<arrow::dataset::ParquetFileFormat>();

After setting up the fs::FileSystem, fs::FileSelector, options, and file format, we can make that dataset::FileSystemDatasetFactory. This simply requires passing in everything we’ve prepared and assigning that to a variable:

  // Now, we get a factory that will let us get our dataset -- we don't have the
  // dataset yet!
  ARROW_ASSIGN_OR_RAISE(auto factory, arrow::dataset::FileSystemDatasetFactory::Make(
                                          fs, selector, read_format, options));

Build Dataset using Factory

With a dataset::FileSystemDatasetFactory set up, we can actually build our dataset::Dataset with dataset::FileSystemDatasetFactory::Finish(), just like with an ArrayBuilder back in the basic tutorial:

  // Now we build our dataset from the factory.
  ARROW_ASSIGN_OR_RAISE(auto read_dataset, factory->Finish());

Now, we have a dataset::Dataset object in memory. This does not mean that the entire dataset is manifested in memory, but that we now have access to tools that allow us to explore and use the dataset that is on disk. For example, we can grab the fragments (files) that make up our whole dataset, and print those out, along with some small info:

  // Print out the fragments
  ARROW_ASSIGN_OR_RAISE(auto fragments, read_dataset->GetFragments());
  for (const auto& fragment : fragments) {
    std::cout << "Found fragment: " << (*fragment)->ToString() << std::endl;
    std::cout << "Partition expression: "
              << (*fragment)->partition_expression().ToString() << std::endl;
  }

Move Dataset into Table

One way we can do something with Datasets is getting them into a Table, where we can do anything we’ve learned we can do to Tables to that Table.

See also

Acero: A C++ streaming execution engine for execution that avoids manifesting the entire dataset in memory.

In order to move a Dataset’s contents into a Table, we need a dataset::Scanner, which scans the data and outputs it to the Table. First, we get a dataset::ScannerBuilder from the dataset::Dataset:

  // Scan dataset into a Table -- once this is done, you can do
  // normal table things with it, like computation and printing. However, now you're
  // also dedicated to being in memory.
  ARROW_ASSIGN_OR_RAISE(auto read_scan_builder, read_dataset->NewScan());

Of course, a Builder’s only use is to get us our dataset::Scanner, so let’s use dataset::ScannerBuilder::Finish():

  ARROW_ASSIGN_OR_RAISE(auto read_scanner, read_scan_builder->Finish());

Now that we have a tool to move through our dataset::Dataset, let’s use it to get our Table. dataset::Scanner::ToTable() offers exactly what we’re looking for, and we can print the results:

  ARROW_ASSIGN_OR_RAISE(std::shared_ptr<arrow::Table> table, read_scanner->ToTable());
  std::cout << table->ToString();

This leaves us with a normal Table. Again, to do things with Datasets without moving to a Table, consider using Acero.

Writing a Dataset to Disk from Table

Writing a dataset::Dataset is a distinct task from writing a single file. The task takes more work than writing a single file, due to needing to be able to parse handle a partitioning scheme across multiple files and folders. This process can be broken up into the following steps:

  1. Prepare a TableBatchReader

  2. Create a dataset::Scanner to pull data from TableBatchReader

  3. Prepare schema, partitioning, and file format options

  4. Set up dataset::FileSystemDatasetWriteOptions – a struct that configures our writing functions

  5. Write dataset to disk

Prepare Data from Table for Writing

We have a Table, and we want to get a dataset::Dataset on disk. In fact, for the sake of exploration, we’ll use a different partitioning scheme for the dataset – instead of just breaking into halves like the original fragments, we’ll partition based on each row’s value in the “a” column.

To get started on that, let’s get a TableBatchReader! This makes it very easy to write to a Dataset, and can be used elsewhere whenever a Table needs to be broken into a stream of RecordBatches. Here, we can just use the TableBatchReader’s constructor, with our table:

  // Now, let's get a table out to disk as a dataset!
  // We make a RecordBatchReader from our Table, then set up a scanner, which lets us
  // go to a file.
  std::shared_ptr<arrow::TableBatchReader> write_dataset =
      std::make_shared<arrow::TableBatchReader>(table);

Create Scanner for Moving Table Data

The process for writing a dataset::Dataset, once a source of data is available, is similar to the reverse of reading it. Before, we used a dataset::Scanner in order to scan into a Table – now, we need one to read out of our TableBatchReader. To get that dataset::Scanner, we’ll make a dataset::ScannerBuilder based on our TableBatchReader, then use that Builder to build a dataset::Scanner:

  auto write_scanner_builder =
      arrow::dataset::ScannerBuilder::FromRecordBatchReader(write_dataset);
  ARROW_ASSIGN_OR_RAISE(auto write_scanner, write_scanner_builder->Finish())

Prepare Schema, Partitioning, and File Format Variables

Since we want to partition based on the “a” column, we need to declare that. When defining our partitioning Schema, we’ll just have a single Field that contains “a”:

  // The partition schema determines which fields are used as keys for partitioning.
  auto partition_schema = arrow::schema({arrow::field("a", arrow::utf8())});

This Schema determines what the key is for partitioning, but we need to choose the algorithm that’ll do something with this key. We will use Hive-style again, this time with our schema passed to it as configuration:

  // We'll use Hive-style partitioning, which creates directories with "key=value"
  // pairs.
  auto partitioning =
      std::make_shared<arrow::dataset::HivePartitioning>(partition_schema);

Several file formats are available, but Parquet is commonly used with Arrow, so we’ll write back out to that:

  // Now, we declare we'll be writing Parquet files.
  auto write_format = std::make_shared<arrow::dataset::ParquetFileFormat>();

Configure FileSystemDatasetWriteOptions

In order to write to disk, we need some configuration. We’ll do so via setting values in a dataset::FileSystemDatasetWriteOptions struct. We’ll initialize it with defaults where possible:

  // This time, we make Options for writing, but do much more configuration.
  arrow::dataset::FileSystemDatasetWriteOptions write_options;
  // Defaults to start.
  write_options.file_write_options = write_format->DefaultWriteOptions();

One important step in writing to file is having a fs::FileSystem to target. Luckily, we have one from when we set it up for reading. This is a simple variable assignment:

  // Use the filesystem we already have.
  write_options.filesystem = fs;

Arrow can make the directory, but it does need a name for said directory, so let’s give it one, call it “write_dataset”:

  // Write to the folder "write_dataset" in current directory.
  write_options.base_dir = "write_dataset";

We made a partitioning method previously, declaring that we’d use Hive-style – this is where we actually pass that to our writing function:

  // Use the partitioning declared above.
  write_options.partitioning = partitioning;

Part of what’ll happen is Arrow will break up files, thus preventing them from being too large to handle. This is what makes a dataset fragmented in the first place. In order to set this up, we need a base name for each fragment in a directory – in this case, we’ll have “part{i}.parquet”, which means the third file (within the same directory) will be called “part3.parquet”, for example:

  // Define what the name for the files making up the dataset will be.
  write_options.basename_template = "part{i}.parquet";

Sometimes, data will be written to the same location more than once, and overwriting will be accepted. Since we may want to run this application more than once, we will set Arrow to overwrite existing data – if we didn’t, Arrow would abort due to seeing existing data after the first run of this application:

  // Set behavior to overwrite existing data -- specifically, this lets this example
  // be run more than once, and allows whatever code you have to overwrite what's there.
  write_options.existing_data_behavior =
      arrow::dataset::ExistingDataBehavior::kOverwriteOrIgnore;

Write Dataset to Disk

Once the dataset::FileSystemDatasetWriteOptions has been configured, and a dataset::Scanner is prepared to parse the data, we can pass the Options and dataset::Scanner to the dataset::FileSystemDataset::Write() to write out to disk:

  // Write to disk!
  ARROW_RETURN_NOT_OK(
      arrow::dataset::FileSystemDataset::Write(write_options, write_scanner));

You can review your disk to see that you’ve written a folder containing subfolders for every value of “a”, which each have Parquet files!

Ending Program

At the end, we just return Status::OK(), so the main() knows that we’re done, and that everything’s okay, just like the preceding tutorials.

  return arrow::Status::OK();
}

With that, you’ve read and written partitioned datasets! This method, with some configuration, will work for any supported dataset format. For an example of such a dataset, the NYC Taxi dataset is a well-known one, which you can find here. Now you can get larger-than-memory data mapped for use!

Which means that now we have to be able to process this data without pulling it all into memory at once. For this, try Acero.

See also

Acero: A C++ streaming execution engine for more information on Acero.

Refer to the below for a copy of the complete code:

 19// (Doc section: Includes)
 20#include <arrow/api.h>
 21#include <arrow/dataset/api.h>
 22// We use Parquet headers for setting up examples; they are not required for using
 23// datasets.
 24#include <parquet/arrow/reader.h>
 25#include <parquet/arrow/writer.h>
 26
 27#include <iostream>
 28// (Doc section: Includes)
 29
 30// (Doc section: Helper Functions)
 31// Generate some data for the rest of this example.
 32arrow::Result<std::shared_ptr<arrow::Table>> CreateTable() {
 33  // This code should look familiar from the basic Arrow example, and is not the
 34  // focus of this example. However, we need data to work on it, and this makes that!
 35  auto schema =
 36      arrow::schema({arrow::field("a", arrow::int64()), arrow::field("b", arrow::int64()),
 37                     arrow::field("c", arrow::int64())});
 38  std::shared_ptr<arrow::Array> array_a;
 39  std::shared_ptr<arrow::Array> array_b;
 40  std::shared_ptr<arrow::Array> array_c;
 41  arrow::NumericBuilder<arrow::Int64Type> builder;
 42  ARROW_RETURN_NOT_OK(builder.AppendValues({0, 1, 2, 3, 4, 5, 6, 7, 8, 9}));
 43  ARROW_RETURN_NOT_OK(builder.Finish(&array_a));
 44  builder.Reset();
 45  ARROW_RETURN_NOT_OK(builder.AppendValues({9, 8, 7, 6, 5, 4, 3, 2, 1, 0}));
 46  ARROW_RETURN_NOT_OK(builder.Finish(&array_b));
 47  builder.Reset();
 48  ARROW_RETURN_NOT_OK(builder.AppendValues({1, 2, 1, 2, 1, 2, 1, 2, 1, 2}));
 49  ARROW_RETURN_NOT_OK(builder.Finish(&array_c));
 50  return arrow::Table::Make(schema, {array_a, array_b, array_c});
 51}
 52
 53// Set up a dataset by writing two Parquet files.
 54arrow::Result<std::string> CreateExampleParquetDataset(
 55    const std::shared_ptr<arrow::fs::FileSystem>& filesystem,
 56    const std::string& root_path) {
 57  // Much like CreateTable(), this is utility that gets us the dataset we'll be reading
 58  // from. Don't worry, we also write a dataset in the example proper.
 59  auto base_path = root_path + "parquet_dataset";
 60  ARROW_RETURN_NOT_OK(filesystem->CreateDir(base_path));
 61  // Create an Arrow Table
 62  ARROW_ASSIGN_OR_RAISE(auto table, CreateTable());
 63  // Write it into two Parquet files
 64  ARROW_ASSIGN_OR_RAISE(auto output,
 65                        filesystem->OpenOutputStream(base_path + "/data1.parquet"));
 66  ARROW_RETURN_NOT_OK(parquet::arrow::WriteTable(
 67      *table->Slice(0, 5), arrow::default_memory_pool(), output, 2048));
 68  ARROW_ASSIGN_OR_RAISE(output,
 69                        filesystem->OpenOutputStream(base_path + "/data2.parquet"));
 70  ARROW_RETURN_NOT_OK(parquet::arrow::WriteTable(
 71      *table->Slice(5), arrow::default_memory_pool(), output, 2048));
 72  return base_path;
 73}
 74
 75arrow::Status PrepareEnv() {
 76  // Get our environment prepared for reading, by setting up some quick writing.
 77  ARROW_ASSIGN_OR_RAISE(auto src_table, CreateTable())
 78  std::shared_ptr<arrow::fs::FileSystem> setup_fs;
 79  // Note this operates in the directory the executable is built in.
 80  char setup_path[256];
 81  char* result = getcwd(setup_path, 256);
 82  if (result == NULL) {
 83    return arrow::Status::IOError("Fetching PWD failed.");
 84  }
 85
 86  ARROW_ASSIGN_OR_RAISE(setup_fs, arrow::fs::FileSystemFromUriOrPath(setup_path));
 87  ARROW_ASSIGN_OR_RAISE(auto dset_path, CreateExampleParquetDataset(setup_fs, ""));
 88
 89  return arrow::Status::OK();
 90}
 91// (Doc section: Helper Functions)
 92
 93// (Doc section: RunMain)
 94arrow::Status RunMain() {
 95  // (Doc section: RunMain)
 96  // (Doc section: PrepareEnv)
 97  ARROW_RETURN_NOT_OK(PrepareEnv());
 98  // (Doc section: PrepareEnv)
 99
100  // (Doc section: FileSystem Declare)
101  // First, we need a filesystem object, which lets us interact with our local
102  // filesystem starting at a given path. For the sake of simplicity, that'll be
103  // the current directory.
104  std::shared_ptr<arrow::fs::FileSystem> fs;
105  // (Doc section: FileSystem Declare)
106
107  // (Doc section: FileSystem Init)
108  // Get the CWD, use it to make the FileSystem object.
109  char init_path[256];
110  char* result = getcwd(init_path, 256);
111  if (result == NULL) {
112    return arrow::Status::IOError("Fetching PWD failed.");
113  }
114  ARROW_ASSIGN_OR_RAISE(fs, arrow::fs::FileSystemFromUriOrPath(init_path));
115  // (Doc section: FileSystem Init)
116
117  // (Doc section: FileSelector Declare)
118  // A file selector lets us actually traverse a multi-file dataset.
119  arrow::fs::FileSelector selector;
120  // (Doc section: FileSelector Declare)
121  // (Doc section: FileSelector Config)
122  selector.base_dir = "parquet_dataset";
123  // Recursive is a safe bet if you don't know the nesting of your dataset.
124  selector.recursive = true;
125  // (Doc section: FileSelector Config)
126  // (Doc section: FileSystemFactoryOptions)
127  // Making an options object lets us configure our dataset reading.
128  arrow::dataset::FileSystemFactoryOptions options;
129  // We'll use Hive-style partitioning. We'll let Arrow Datasets infer the partition
130  // schema. We won't set any other options, defaults are fine.
131  options.partitioning = arrow::dataset::HivePartitioning::MakeFactory();
132  // (Doc section: FileSystemFactoryOptions)
133  // (Doc section: File Format Setup)
134  auto read_format = std::make_shared<arrow::dataset::ParquetFileFormat>();
135  // (Doc section: File Format Setup)
136  // (Doc section: FileSystemDatasetFactory Make)
137  // Now, we get a factory that will let us get our dataset -- we don't have the
138  // dataset yet!
139  ARROW_ASSIGN_OR_RAISE(auto factory, arrow::dataset::FileSystemDatasetFactory::Make(
140                                          fs, selector, read_format, options));
141  // (Doc section: FileSystemDatasetFactory Make)
142  // (Doc section: FileSystemDatasetFactory Finish)
143  // Now we build our dataset from the factory.
144  ARROW_ASSIGN_OR_RAISE(auto read_dataset, factory->Finish());
145  // (Doc section: FileSystemDatasetFactory Finish)
146  // (Doc section: Dataset Fragments)
147  // Print out the fragments
148  ARROW_ASSIGN_OR_RAISE(auto fragments, read_dataset->GetFragments());
149  for (const auto& fragment : fragments) {
150    std::cout << "Found fragment: " << (*fragment)->ToString() << std::endl;
151    std::cout << "Partition expression: "
152              << (*fragment)->partition_expression().ToString() << std::endl;
153  }
154  // (Doc section: Dataset Fragments)
155  // (Doc section: Read Scan Builder)
156  // Scan dataset into a Table -- once this is done, you can do
157  // normal table things with it, like computation and printing. However, now you're
158  // also dedicated to being in memory.
159  ARROW_ASSIGN_OR_RAISE(auto read_scan_builder, read_dataset->NewScan());
160  // (Doc section: Read Scan Builder)
161  // (Doc section: Read Scanner)
162  ARROW_ASSIGN_OR_RAISE(auto read_scanner, read_scan_builder->Finish());
163  // (Doc section: Read Scanner)
164  // (Doc section: To Table)
165  ARROW_ASSIGN_OR_RAISE(std::shared_ptr<arrow::Table> table, read_scanner->ToTable());
166  std::cout << table->ToString();
167  // (Doc section: To Table)
168
169  // (Doc section: TableBatchReader)
170  // Now, let's get a table out to disk as a dataset!
171  // We make a RecordBatchReader from our Table, then set up a scanner, which lets us
172  // go to a file.
173  std::shared_ptr<arrow::TableBatchReader> write_dataset =
174      std::make_shared<arrow::TableBatchReader>(table);
175  // (Doc section: TableBatchReader)
176  // (Doc section: WriteScanner)
177  auto write_scanner_builder =
178      arrow::dataset::ScannerBuilder::FromRecordBatchReader(write_dataset);
179  ARROW_ASSIGN_OR_RAISE(auto write_scanner, write_scanner_builder->Finish())
180  // (Doc section: WriteScanner)
181  // (Doc section: Partition Schema)
182  // The partition schema determines which fields are used as keys for partitioning.
183  auto partition_schema = arrow::schema({arrow::field("a", arrow::utf8())});
184  // (Doc section: Partition Schema)
185  // (Doc section: Partition Create)
186  // We'll use Hive-style partitioning, which creates directories with "key=value"
187  // pairs.
188  auto partitioning =
189      std::make_shared<arrow::dataset::HivePartitioning>(partition_schema);
190  // (Doc section: Partition Create)
191  // (Doc section: Write Format)
192  // Now, we declare we'll be writing Parquet files.
193  auto write_format = std::make_shared<arrow::dataset::ParquetFileFormat>();
194  // (Doc section: Write Format)
195  // (Doc section: Write Options)
196  // This time, we make Options for writing, but do much more configuration.
197  arrow::dataset::FileSystemDatasetWriteOptions write_options;
198  // Defaults to start.
199  write_options.file_write_options = write_format->DefaultWriteOptions();
200  // (Doc section: Write Options)
201  // (Doc section: Options FS)
202  // Use the filesystem we already have.
203  write_options.filesystem = fs;
204  // (Doc section: Options FS)
205  // (Doc section: Options Target)
206  // Write to the folder "write_dataset" in current directory.
207  write_options.base_dir = "write_dataset";
208  // (Doc section: Options Target)
209  // (Doc section: Options Partitioning)
210  // Use the partitioning declared above.
211  write_options.partitioning = partitioning;
212  // (Doc section: Options Partitioning)
213  // (Doc section: Options Name Template)
214  // Define what the name for the files making up the dataset will be.
215  write_options.basename_template = "part{i}.parquet";
216  // (Doc section: Options Name Template)
217  // (Doc section: Options File Behavior)
218  // Set behavior to overwrite existing data -- specifically, this lets this example
219  // be run more than once, and allows whatever code you have to overwrite what's there.
220  write_options.existing_data_behavior =
221      arrow::dataset::ExistingDataBehavior::kOverwriteOrIgnore;
222  // (Doc section: Options File Behavior)
223  // (Doc section: Write Dataset)
224  // Write to disk!
225  ARROW_RETURN_NOT_OK(
226      arrow::dataset::FileSystemDataset::Write(write_options, write_scanner));
227  // (Doc section: Write Dataset)
228  // (Doc section: Ret)
229  return arrow::Status::OK();
230}
231// (Doc section: Ret)
232// (Doc section: Main)
233int main() {
234  arrow::Status st = RunMain();
235  if (!st.ok()) {
236    std::cerr << st << std::endl;
237    return 1;
238  }
239  return 0;
240}
241// (Doc section: Main)

previous

Arrow Compute

next

User Guide