pyarrow.csv.CSVStreamingReader

class pyarrow.csv.CSVStreamingReader

Bases: RecordBatchReader

An object that reads record batches incrementally from a CSV file.

Should not be instantiated directly by user code.

__init__(*args, **kwargs)

Methods

__init__(*args, **kwargs)
close(self)	Release any resources associated with the reader.
from_batches(Schema schema, batches)	Create RecordBatchReader from an iterable of batches.
iter_batches_with_custom_metadata(self)	Iterate over record batches from the stream along with their custom metadata.
read_all(self)	Read all record batches as a pyarrow.Table.
read_next_batch(self)	Read next RecordBatch from the stream.
read_next_batch_with_custom_metadata(self)	Read next RecordBatch from the stream along with its custom metadata.
read_pandas(self, **options)	Read contents of stream to a pandas.DataFrame.

Attributes

schema

close(self)

Release any resources associated with the reader.

static from_batches(Schema schema, batches)

Create RecordBatchReader from an iterable of batches.

Parameters:

schemaSchema

The shared schema of the record batches

batchesIterable[RecordBatch]

The batches that this reader will return.

Returns:

readerRecordBatchReader

iter_batches_with_custom_metadata(self)

Iterate over record batches from the stream along with their custom metadata.

Yields:

RecordBatchWithMetadata

read_all(self)

Read all record batches as a pyarrow.Table.

Returns:

Table

read_next_batch(self)

Read next RecordBatch from the stream.

Returns:

RecordBatch

Raises:

StopIteration:

At end of stream.

read_next_batch_with_custom_metadata(self)

Read next RecordBatch from the stream along with its custom metadata.

Returns:

batchRecordBatch

custom_metadataKeyValueMetadata

Raises:

StopIteration:

At end of stream.

read_pandas(self, **options)

Read contents of stream to a pandas.DataFrame.

Read all record batches as a pyarrow.Table then convert it to a pandas.DataFrame using Table.to_pandas.

Parameters:

**options

Arguments to forward to Table.to_pandas().

Returns:

dfpandas.DataFrame

schema