File system classes

File system classes

Functions

Properties

char * base-name Read
char * dir-name Read
char * extension Read
gint64 mtime Read / Write
char * path Read / Write
gint64 size Read / Write
GArrowFileType type Read / Write
gboolean allow-not-found Read / Write
char * base-dir Read / Write
int max-recursion Read / Write
gboolean recursive Read / Write
gpointer file-system Write / Construct Only
GArrowS3LogLevel log-level Read / Write / Construct
GArrowFileSystem * base-file-system Read / Write / Construct Only
GArrowFileSystem * base-file-system Read / Write / Construct Only

Types and Values

Object Hierarchy

    GEnum
    ├── GArrowFileType
    ╰── GArrowS3LogLevel
    GObject
    ├── GArrowFileInfo
    ├── GArrowFileSelector
    ├── GArrowFileSystem
       ├── GArrowGCSFileSystem
       ├── GArrowHDFSFileSystem
       ├── GArrowLocalFileSystem
       ├── GArrowMockFileSystem
       ├── GArrowS3FileSystem
       ├── GArrowSlowFileSystem
       ╰── GArrowSubTreeFileSystem
    ╰── GArrowS3GlobalOptions

Includes

#include <arrow-glib/arrow-glib.h>

Description

GArrowFileInfo is a class for information for a file system entry.

GArrowFileSelector is a class for a selector for file system APIs.

GArrowFileSystem is an interface for file system.

GArrowSubTreeFileSystem is a delegator to another file system that is a logical view of a subtree of a file system, such as a directory in a local file system.

GArrowSlowFileSystem is a delegator to another file system. This inserts latencies at various points.

GArrowMockFileSystem is a class for mock file system that holds its contents in memory.

GArrowHDFSFileSystem is a class for HDFS-backed file system.

GArrowS3GlobalOptions is a class for options to initialize S3 APIs.

GArrowS3FileSystem is a class for S3-backed file system.

GArrowGCSFileSystem is a class for GCS-backed file system.

Functions

garrow_file_info_new ()

GArrowFileInfo *
garrow_file_info_new (void);

Returns

A newly created GArrowFileInfo.

Since: 0.17.0


garrow_file_info_equal ()

gboolean
garrow_file_info_equal (GArrowFileInfo *file_info,
                        GArrowFileInfo *other_file_info);

Parameters

file_info

A GArrowFileInfo.

 

other_file_info

A GArrowFileInfo to be compared.

 

Returns

TRUE if both of them have the same data, FALSE otherwise.

Since: 0.17.0


garrow_file_info_is_file ()

gboolean
garrow_file_info_is_file (GArrowFileInfo *file_info);

Parameters

file_info

A GArrowFileInfo.

 

Returns

TRUE if the entry is a file, FALSE otherwise.

Since: 0.17.0


garrow_file_info_is_dir ()

gboolean
garrow_file_info_is_dir (GArrowFileInfo *file_info);

Parameters

file_info

A GArrowFileInfo.

 

Returns

TRUE if the entry is a directory, FALSE otherwise.

Since: 0.17.0


garrow_file_info_to_string ()

gchar *
garrow_file_info_to_string (GArrowFileInfo *file_info);

Parameters

file_info

A GArrowFileInfo.

 

Returns

The string representation of the file statistics.

It should be freed with g_free() when no longer needed.

Since: 0.17.0


garrow_file_system_create ()

GArrowFileSystem *
garrow_file_system_create (const gchar *uri,
                           GError **error);

This is a factory function to create a specific GArrowFileSystem object.

Parameters

uri

An URI to specify file system with options. If you only have an absolute path, g_filename_to_uri() will help you.

 

error

Return location for a GError or NULL.

[nullable]

Returns

The newly created file system that is an object of a subclass of GArrowFileSystem.

[nullable][transfer full]

Since: 3.0.0


garrow_file_system_get_type_name ()

gchar *
garrow_file_system_get_type_name (GArrowFileSystem *file_system);

Parameters

file_system

A GArrowFileSystem.

 

Returns

The name of file system type.

It should be freed with g_free() when no longer needed.

Since: 0.17.0


garrow_file_system_get_file_info ()

GArrowFileInfo *
garrow_file_system_get_file_info (GArrowFileSystem *file_system,
                                  const gchar *path,
                                  GError **error);

Get information for the given target.

Any symlink is automatically dereferenced, recursively. A non-existing or unreachable file returns an OK status and has a GArrowFileType of value GARROW_FILE_TYPE_NOT_FOUND. An error status indicates a truly exceptional condition (low-level I/O error, etc.).

Parameters

file_system

A GArrowFileSystem.

 

path

The path of the target.

 

error

Return location for a GError or NULL.

[nullable]

Returns

A GArrowFileInfo.

[nullable][transfer full]

Since: 0.17.0


garrow_file_system_get_file_infos_paths ()

GList *
garrow_file_system_get_file_infos_paths
                               (GArrowFileSystem *file_system,
                                const gchar **paths,
                                gsize n_paths,
                                GError **error);

Get information same as garrow_file_system_get_file_info() for the given many targets at once.

Parameters

file_system

A GArrowFileSystem.

 

paths

The paths of the targets.

[array length=n_paths]

n_paths

The number of items in paths .

 

error

Return location for a GError or NULL.

[nullable]

Returns

A list of GArrowFileInfo.

[element-type GArrowFileInfo][transfer full]

Since: 0.17.0


garrow_file_system_get_file_infos_selector ()

GList *
garrow_file_system_get_file_infos_selector
                               (GArrowFileSystem *file_system,
                                GArrowFileSelector *file_selector,
                                GError **error);

Get information same as garrow_file_system_get_file_info() according to a selector.

The selector's base directory will not be part of the results, even if it exists.

Parameters

file_system

A GArrowFileSystem.

 

file_selector

A GArrowFileSelector.

 

error

Return location for a GError or NULL.

[nullable]

Returns

A list of GArrowFileInfo.

[element-type GArrowFileInfo][transfer full]

Since: 0.17.0


garrow_file_system_create_dir ()

gboolean
garrow_file_system_create_dir (GArrowFileSystem *file_system,
                               const gchar *path,
                               gboolean recursive,
                               GError **error);

Create a directory and subdirectories. This function succeeds if the directory already exists.

Parameters

file_system

A GArrowFileSystem.

 

path

The paths of the directory.

 

recursive

Whether creating directory recursively or not.

 

error

Return location for a GError or NULL.

[nullable]

Returns

TRUE on success, FALSE if there was an error.

Since: 0.17.0


garrow_file_system_delete_dir ()

gboolean
garrow_file_system_delete_dir (GArrowFileSystem *file_system,
                               const gchar *path,
                               GError **error);

Delete a directory and its contents, recursively.

Parameters

file_system

A GArrowFileSystem.

 

path

The paths of the directory.

 

error

Return location for a GError or NULL.

[nullable]

Returns

TRUE on success, FALSE if there was an error.

Since: 0.17.0


garrow_file_system_delete_dir_contents ()

gboolean
garrow_file_system_delete_dir_contents
                               (GArrowFileSystem *file_system,
                                const gchar *path,
                                GError **error);

Delete a directory's contents, recursively. Like garrow_file_system_delete_dir(), but doesn't delete the directory itself. Passing an empty path ("") will wipe the entire file system tree.

Parameters

file_system

A GArrowFileSystem.

 

path

The paths of the directory.

 

error

Return location for a GError or NULL.

[nullable]

Returns

TRUE on success, FALSE if there was an error.

Since: 0.17.0


garrow_file_system_delete_file ()

gboolean
garrow_file_system_delete_file (GArrowFileSystem *file_system,
                                const gchar *path,
                                GError **error);

Delete a file.

Parameters

file_system

A GArrowFileSystem.

 

path

The paths of the file to be delete.

 

error

Return location for a GError or NULL.

[nullable]

Returns

TRUE on success, FALSE if there was an error.

Since: 0.17.0


garrow_file_system_delete_files ()

gboolean
garrow_file_system_delete_files (GArrowFileSystem *file_system,
                                 const gchar **paths,
                                 gsize n_paths,
                                 GError **error);

Delete many files.

Parameters

file_system

A GArrowFileSystem.

 

paths

The paths of the files to be delete.

[array length=n_paths]

n_paths

The number of items in paths .

 

error

Return location for a GError or NULL.

[nullable]

Returns

TRUE on success, FALSE if there was an error.

Since: 0.17.0


garrow_file_system_move ()

gboolean
garrow_file_system_move (GArrowFileSystem *file_system,
                         const gchar *src,
                         const gchar *dest,
                         GError **error);

Move / rename a file or a directory. If the destination exists:

  • if it is a non-empty directory, an error is returned

  • otherwise, if it has the same type as the source, it is replaced

  • otherwise, behavior is unspecified (implementation-dependent).

Parameters

file_system

A GArrowFileSystem.

 

src

The path of the source file.

 

dest

The path of the destination.

 

error

Return location for a GError or NULL.

[nullable]

Returns

TRUE on success, FALSE if there was an error.

Since: 0.17.0


garrow_file_system_copy_file ()

gboolean
garrow_file_system_copy_file (GArrowFileSystem *file_system,
                              const gchar *src,
                              const gchar *dest,
                              GError **error);

Copy a file. If the destination exists and is a directory, an error is returned. Otherwise, it is replaced.

Parameters

file_system

A GArrowFileSystem.

 

src

The path of the source file.

 

dest

The path of the destination.

 

error

Return location for a GError or NULL.

[nullable]

Returns

TRUE on success, FALSE if there was an error.

Since: 0.17.0


garrow_file_system_open_input_stream ()

GArrowInputStream *
garrow_file_system_open_input_stream (GArrowFileSystem *file_system,
                                      const gchar *path,
                                      GError **error);

Open an input stream for sequential reading.

Parameters

file_system

A GArrowFileSystem.

 

path

The path of the input stream.

 

error

Return location for a GError or NULL.

[nullable]

Returns

A newly created GArrowInputStream.

[nullable][transfer full]

Since: 0.17.0


garrow_file_system_open_input_file ()

GArrowSeekableInputStream *
garrow_file_system_open_input_file (GArrowFileSystem *file_system,
                                    const gchar *path,
                                    GError **error);

Open an input file for random access reading.

Parameters

file_system

A GArrowFileSystem.

 

path

The path of the input file.

 

error

Return location for a GError or NULL.

[nullable]

Returns

A newly created GArrowSeekableInputStream.

[nullable][transfer full]

Since: 0.17.0


garrow_file_system_open_output_stream ()

GArrowOutputStream *
garrow_file_system_open_output_stream (GArrowFileSystem *file_system,
                                       const gchar *path,
                                       GError **error);

Open an output stream for sequential writing. If the target already exists, the existing data is truncated.

Parameters

file_system

A GArrowFileSystem.

 

path

The path of the output stream.

 

error

Return location for a GError or NULL.

[nullable]

Returns

A newly created GArrowOutputStream.

[nullable][transfer full]

Since: 0.17.0


garrow_file_system_open_append_stream ()

GArrowOutputStream *
garrow_file_system_open_append_stream (GArrowFileSystem *file_system,
                                       const gchar *path,
                                       GError **error);

Open an output stream for appending. If the target doesn't exist, a new empty file is created.

Parameters

file_system

A GArrowFileSystem.

 

path

The path of the output stream.

 

error

Return location for a GError or NULL.

[nullable]

Returns

A newly created GArrowOutputStream for appending.

[nullable][transfer full]

Since: 0.17.0


garrow_sub_tree_file_system_new ()

GArrowSubTreeFileSystem *
garrow_sub_tree_file_system_new (const gchar *base_path,
                                 GArrowFileSystem *base_file_system);

Parameters

base_path

A base path of the sub tree file system.

 

base_file_system

A GArrowFileSystem as the base file system.

 

Returns

A newly created GArrowSubTreeFileSystem.

[transfer full]

Since: 0.17.0


garrow_slow_file_system_new_average_latency ()

GArrowSlowFileSystem *
garrow_slow_file_system_new_average_latency
                               (GArrowFileSystem *base_file_system,
                                gdouble average_latency);

The latency is normally distributed with a standard deviation of average_latency * 0.1.

The random seed is given by the default random device.

Parameters

base_file_system

A GArrowFileSystem as the base file system.

 

average_latency

The average value of the latency.

 

Returns

A newly created GArrowSlowFileSystem.

[transfer full]

Since: 0.17.0


garrow_slow_file_system_new_average_latency_and_seed ()

GArrowSlowFileSystem *
garrow_slow_file_system_new_average_latency_and_seed
                               (GArrowFileSystem *base_file_system,
                                gdouble average_latency,
                                gint32 seed);

The latency is normally distributed with a standard deviation of average_latency * 0.1.

Parameters

base_file_system

A GArrowFileSystem as the base file system.

 

average_latency

The average value of the latency.

 

seed

A random seed.

 

Returns

A newly created GArrowSlowFileSystem.

[transfer full]

Since: 0.17.0


garrow_s3_global_options_new ()

GArrowS3GlobalOptions *
garrow_s3_global_options_new (void);

Returns

A newly created GArrowS3GlobalOptions.

Since: 7.0.0


garrow_s3_is_enabled ()

gboolean
garrow_s3_is_enabled (void);

Returns

TRUE if Apache Arrow C++ is built with S3 support, FALSE otherwise.

Since: 7.0.0


garrow_s3_initialize ()

gboolean
garrow_s3_initialize (GArrowS3GlobalOptions *options,
                      GError **error);

Normally, you don't need to call this function because the S3 APIs are initialized with the default options automatically. If you want to call this function, you must call this function before you use any GArrowS3FileSystem related APIs.

Parameters

options

Options to initialize the S3 APIs.

[nullable]

error

Return location for a GError or NULL.

[nullable]

Returns

TRUE on success, FALSE on error.

Since: 7.0.0


garrow_s3_finalize ()

gboolean
garrow_s3_finalize (GError **error);

Finalize the S3 APIs.

Parameters

error

Return location for a GError or NULL.

[nullable]

Returns

TRUE on success, FALSE on error.

Since: 7.0.0

Types and Values

GArrowTimePoint

typedef gint64 GArrowTimePoint;

enum GArrowFileType

They are corresponding to arrow::fs::FileType values.

Members

GARROW_FILE_TYPE_NOT_FOUND

Entry is not found

 

GARROW_FILE_TYPE_UNKNOWN

Entry exists but its type is unknown

 

GARROW_FILE_TYPE_FILE

Entry is a regular file

 

GARROW_FILE_TYPE_DIR

Entry is a directory

 

Since: 1.0.0


GARROW_TYPE_FILE_INFO

#define GARROW_TYPE_FILE_INFO (garrow_file_info_get_type())

struct GArrowFileInfoClass

struct GArrowFileInfoClass {
  GObjectClass parent_class;
};

GARROW_TYPE_FILE_SELECTOR

#define GARROW_TYPE_FILE_SELECTOR (garrow_file_selector_get_type())

struct GArrowFileSelectorClass

struct GArrowFileSelectorClass {
  GObjectClass parent_class;
};

GARROW_TYPE_FILE_SYSTEM

#define GARROW_TYPE_FILE_SYSTEM (garrow_file_system_get_type())

struct GArrowFileSystemClass

struct GArrowFileSystemClass {
  GObjectClass parent_class;
};

GARROW_TYPE_SUB_TREE_FILE_SYSTEM

#define GARROW_TYPE_SUB_TREE_FILE_SYSTEM (garrow_sub_tree_file_system_get_type())

struct GArrowSubTreeFileSystemClass

struct GArrowSubTreeFileSystemClass {
  GArrowFileSystemClass parent_class;
};

GARROW_TYPE_SLOW_FILE_SYSTEM

#define GARROW_TYPE_SLOW_FILE_SYSTEM (garrow_slow_file_system_get_type())

struct GArrowSlowFileSystemClass

struct GArrowSlowFileSystemClass {
  GArrowFileSystemClass parent_class;
};

GARROW_TYPE_MOCK_FILE_SYSTEM

#define GARROW_TYPE_MOCK_FILE_SYSTEM (garrow_mock_file_system_get_type())

struct GArrowMockFileSystemClass

struct GArrowMockFileSystemClass {
  GArrowFileSystemClass parent_class;
};

GARROW_TYPE_HDFS_FILE_SYSTEM

#define GARROW_TYPE_HDFS_FILE_SYSTEM (garrow_hdfs_file_system_get_type())

struct GArrowHDFSFileSystemClass

struct GArrowHDFSFileSystemClass {
  GArrowFileSystemClass parent_class;
};

enum GArrowS3LogLevel

They are corresponding to arrow::fs::S3LogLevel values.

Members

GARROW_S3_LOG_LEVEL_OFF

Off.

 

GARROW_S3_LOG_LEVEL_FATAL

Fatal. This is the default.

 

GARROW_S3_LOG_LEVEL_ERROR

Error.

 

GARROW_S3_LOG_LEVEL_WARN

Warn.

 

GARROW_S3_LOG_LEVEL_INFO

Info.

 

GARROW_S3_LOG_LEVEL_DEBUG

Debug.

 

GARROW_S3_LOG_LEVEL_TRACE

Trace.

 

Since: 7.0.0


GARROW_TYPE_S3_GLOBAL_OPTIONS

#define GARROW_TYPE_S3_GLOBAL_OPTIONS (garrow_s3_global_options_get_type())

struct GArrowS3GlobalOptionsClass

struct GArrowS3GlobalOptionsClass {
  GObjectClass parent_class;
};

GARROW_TYPE_S3_FILE_SYSTEM

#define GARROW_TYPE_S3_FILE_SYSTEM (garrow_s3_file_system_get_type())

struct GArrowS3FileSystemClass

struct GArrowS3FileSystemClass {
  GArrowFileSystemClass parent_class;
};

GARROW_TYPE_GCS_FILE_SYSTEM

#define GARROW_TYPE_GCS_FILE_SYSTEM (garrow_gcs_file_system_get_type())

struct GArrowGCSFileSystemClass

struct GArrowGCSFileSystemClass {
  GArrowFileSystemClass parent_class;
};

GArrowFileInfo

typedef struct _GArrowFileInfo GArrowFileInfo;

GArrowFileSelector

typedef struct _GArrowFileSelector GArrowFileSelector;

GArrowFileSystem

typedef struct _GArrowFileSystem GArrowFileSystem;

GArrowGCSFileSystem

typedef struct _GArrowGCSFileSystem GArrowGCSFileSystem;

GArrowHDFSFileSystem

typedef struct _GArrowHDFSFileSystem GArrowHDFSFileSystem;

GArrowMockFileSystem

typedef struct _GArrowMockFileSystem GArrowMockFileSystem;

GArrowS3FileSystem

typedef struct _GArrowS3FileSystem GArrowS3FileSystem;

GArrowS3GlobalOptions

typedef struct _GArrowS3GlobalOptions GArrowS3GlobalOptions;

GArrowSlowFileSystem

typedef struct _GArrowSlowFileSystem GArrowSlowFileSystem;

GArrowSubTreeFileSystem

typedef struct _GArrowSubTreeFileSystem GArrowSubTreeFileSystem;

Property Details

The “base-name” property

  “base-name”                char *

The file base name (component after the last directory separator).

Owner: GArrowFileInfo

Flags: Read

Default value: ""

Since: 0.17.0


The “dir-name” property

  “dir-name”                 char *

The directory base name (component before the file base name).

Owner: GArrowFileInfo

Flags: Read

Default value: ""

Since: 0.17.0


The “extension” property

  “extension”                char *

The file extension (excluding the dot).

Owner: GArrowFileInfo

Flags: Read

Default value: ""

Since: 0.17.0


The “mtime” property

  “mtime”                    gint64

The time of last modification, if available.

Owner: GArrowFileInfo

Flags: Read / Write

Allowed values: >= -1

Default value: -1

Since: 0.17.0


The “path” property

  “path”                     char *

The full file path in the file system.

Owner: GArrowFileInfo

Flags: Read / Write

Default value: ""

Since: 0.17.0


The “size” property

  “size”                     gint64

The size in bytes, if available Only regular files are guaranteed to have a size.

Owner: GArrowFileInfo

Flags: Read / Write

Allowed values: >= -1

Default value: -1

Since: 0.17.0


The “type” property

  “type”                     GArrowFileType

The type of the entry.

Owner: GArrowFileInfo

Flags: Read / Write

Default value: GARROW_FILE_TYPE_UNKNOWN

Since: 0.17.0


The “allow-not-found” property

  “allow-not-found”          gboolean

The behavior if base_dir isn't found in the file system. If false, an error is returned. If true, an empty selection is returned.

Owner: GArrowFileSelector

Flags: Read / Write

Default value: FALSE

Since: 0.17.0


The “base-dir” property

  “base-dir”                 char *

The directory in which to select files. If the path exists but doesn't point to a directory, this should be an error.

Owner: GArrowFileSelector

Flags: Read / Write

Default value: ""

Since: 0.17.0


The “max-recursion” property

  “max-recursion”            int

The maximum number of subdirectories to recurse into.

Owner: GArrowFileSelector

Flags: Read / Write

Allowed values: >= 0

Default value: 2147483647

Since: 0.17.0


The “recursive” property

  “recursive”                gboolean

Whether to recurse into subdirectories.

Owner: GArrowFileSelector

Flags: Read / Write

Default value: FALSE

Since: 0.17.0


The “file-system” property

  “file-system”              gpointer

The raw std::shared<arrow::fs::FileSystem> *.

Owner: GArrowFileSystem

Flags: Write / Construct Only


The “log-level” property

  “log-level”                GArrowS3LogLevel

The log level of S3 APIs.

Owner: GArrowS3GlobalOptions

Flags: Read / Write / Construct

Default value: GARROW_S3_LOG_LEVEL_FATAL

Since: 7.0.0


The “base-file-system” property

  “base-file-system”         GArrowFileSystem *

The base GArrowFileSystem.

Owner: GArrowSlowFileSystem

Flags: Read / Write / Construct Only


The “base-file-system” property

  “base-file-system”         GArrowFileSystem *

The base GArrowFileSystem.

Owner: GArrowSubTreeFileSystem

Flags: Read / Write / Construct Only