pyarrow.fs.copy_files#

pyarrow.fs.copy_files(source, destination, source_filesystem=None, destination_filesystem=None, *, chunk_size=1048576, use_threads=True)[source]#

Copy files between FileSystems.

This functions allows you to recursively copy directories of files from one file system to another, such as from S3 to your local machine.

Parameters:

sourcestr: Source file path or URI to a single file or directory. If a directory, files will be copied recursively from this path.
destinationstr: Destination file path or URI. If source is a file, destination is also interpreted as the destination file (not directory). Directories will be created as necessary.
source_filesystemFileSystem, optional: Source filesystem, needs to be specified if source is not a URI, otherwise inferred.
destination_filesystemFileSystem, optional: Destination filesystem, needs to be specified if destination is not a URI, otherwise inferred.
chunk_sizeint, default 1MB: The maximum size of block to read before flushing to the destination file. A larger chunk_size will use more memory while copying but may help accommodate high latency FileSystems.
use_threadsbool, default True: Whether to use multiple threads to accelerate copying.

Examples

Inspect an S3 bucket’s files:

>>> s3, path = fs.FileSystem.from_uri(
...            "s3://registry.opendata.aws/roda/ndjson/")
>>> selector = fs.FileSelector(path)
>>> s3.get_file_info(selector)
[<FileInfo for 'registry.opendata.aws/roda/ndjson/index.ndjson':...]

Copy one file from S3 bucket to a local directory:

>>> fs.copy_files("s3://registry.opendata.aws/roda/ndjson/index.ndjson",
...               "file:///{}/index_copy.ndjson".format(local_path))

>>> fs.LocalFileSystem().get_file_info(str(local_path)+
...                                    '/index_copy.ndjson')
<FileInfo for '.../index_copy.ndjson': type=FileType.File, size=...>

Copy file using a FileSystem object:

>>> fs.copy_files("registry.opendata.aws/roda/ndjson/index.ndjson",
...               "file:///{}/index_copy.ndjson".format(local_path),
...               source_filesystem=fs.S3FileSystem())