pyarrow.output_stream

pyarrow.output_stream(source, compression='detect', buffer_size=None)

Create an Arrow output stream.

Parameters:

sourcestr, Path, buffer, file-like object

The source to open for writing.

compressionstr optional, default ‘detect’

The compression algorithm to use for on-the-fly compression. If “detect” and source is a file path, then compression will be chosen based on the file extension. If None, no compression will be applied. Otherwise, a well-known algorithm name must be supplied (e.g. “gzip”).

buffer_sizeint, default None

If None or 0, no buffering will happen. Otherwise the size of the temporary write buffer.

Examples

Create a writable NativeFile from a pyarrow Buffer:

>>> import pyarrow as pa
>>> data = b"buffer data"
>>> empty_obj = bytearray(11)
>>> buf = pa.py_buffer(empty_obj)
>>> with pa.output_stream(buf) as stream:
...     stream.write(data)
...
11
>>> with pa.input_stream(buf) as stream:
...     stream.read(6)
...
b'buffer'

or from a memoryview object:

>>> buf = memoryview(empty_obj)
>>> with pa.output_stream(buf) as stream:
...     stream.write(data)
...
11
>>> with pa.input_stream(buf) as stream:
...     stream.read()
...
b'buffer data'

Create a writable NativeFile from a string or file path:

>>> with pa.output_stream('example_second.txt') as stream:
...     stream.write(b'Write some data')
...
15
>>> with pa.input_stream('example_second.txt') as stream:
...     stream.read()
...
b'Write some data'