Module reader 

JSON reader

This JSON reader allows JSON records to be read into the Arrow memory model. Records are loaded in batches and are then converted from the record-oriented representation to the columnar arrow data model.

The reader ignores whitespace between JSON values, including \n and \r, allowing parsing of sequences of one or more arbitrarily formatted JSON values, including but not limited to newline-delimited JSON.

Basic Usage

Reader can be used directly with synchronous data sources, such as std::fs::File

let schema = Arc::new(Schema::new(vec![
    Field::new("a", DataType::Float64, false),
    Field::new("b", DataType::Float64, false),
    Field::new("c", DataType::Boolean, true),
]));

let file = File::open("test/data/basic.json").unwrap();

let mut json = arrow_json::ReaderBuilder::new(schema).build(BufReader::new(file)).unwrap();
let batch = json.next().unwrap().unwrap();

Async Usage

The lower-level Decoder can be integrated with various forms of async data streams, and can be used with any of the async IO primitives found within the Rust ecosystem.

For example, it can be used with an arbitrary Stream of Bytes as shown below

fn decode_stream<S: Stream<Item = Bytes> + Unpin>(
    mut decoder: Decoder,
    mut input: S,
) -> impl Stream<Item = Result<RecordBatch, ArrowError>> {
    let mut buffered = Bytes::new();
    futures::stream::poll_fn(move |cx| {
        loop {
            if buffered.is_empty() {
                buffered = match ready!(input.poll_next_unpin(cx)) {
                    Some(b) => b,
                    None => break,
                };
            }
            let decoded = match decoder.decode(buffered.as_ref()) {
                Ok(decoded) => decoded,
                Err(e) => return Poll::Ready(Some(Err(e))),
            };
            let read = buffered.len();
            buffered.advance(decoded);
            if decoded != read {
                break
            }
        }

        Poll::Ready(decoder.flush().transpose())
    })
}

In a similar vein, it can also be used with tokio-based IO primitives

fn decode_stream<R: AsyncBufRead + Unpin>(
    mut decoder: Decoder,
    mut reader: R,
) -> impl Stream<Item = Result<RecordBatch, ArrowError>> {
    futures::stream::poll_fn(move |cx| {
        loop {
            let b = match ready!(Pin::new(&mut reader).poll_fill_buf(cx)) {
                Ok(b) if b.is_empty() => break,
                Ok(b) => b,
                Err(e) => return Poll::Ready(Some(Err(e.into()))),
            };
            let read = b.len();
            let decoded = match decoder.decode(b) {
                Ok(decoded) => decoded,
                Err(e) => return Poll::Ready(Some(Err(e))),
            };
            Pin::new(&mut reader).consume(decoded);
            if decoded != read {
                break;
            }
        }

        Poll::Ready(decoder.flush().transpose())
    })
}

Customizing the decoder

The decoding of each data type can be customized using ReaderBuilder::with_decoder_factory. For example, you can override the default decoding of binary data from hex strings to decode from an array of integers instead, or provide decoders for types with no built-in decoding. See the example on DecoderFactory.

Modules

binary_array 🔒

boolean_array 🔒

decimal_array 🔒

list_array 🔒

map_array 🔒

null_array 🔒

primitive_array 🔒

run_end_array 🔒

schema 🔒

serializer 🔒

string_array 🔒

string_view_array 🔒

struct_array 🔒

tape 🔒

timestamp_array 🔒

value_iter 🔒

Structs

CheckedDecoder 🔒

Validates the output of a DecoderFactory decoder before it reaches the arrays built from it, some of which are constructed without further checks.

Decoder

A low-level interface for reading JSON data from a byte stream

DecoderContext

Context for decoder creation, containing configuration.

Reader

Reads JSON data with a known schema directly into arrow [RecordBatch]

ReaderBuilder

A builder for Reader and Decoder

Tape

A decoded JSON tape

ValueIter

JSON file reader that produces a serde_json::Value iterator from a Read trait

Enums

TapeElement

We decode JSON to a flattened tape representation, allowing for efficient traversal of the JSON data

Traits

ArrayDecoder

Decodes a column of JSON values from a Tape into an [ArrayRef]

DecoderFactory

A trait to create custom decoders for specific data types.

Functions

infer_json_schema

Infer the fields of a JSON file by reading the first n records of the buffer, with max_read_records controlling the maximum number of records to read.

infer_json_schema_from_iterator

Infer the fields of a JSON file by reading all items from the JSON Value Iterator.

infer_json_schema_from_seekable

Infer the fields of a JSON file by reading the first n records of the file, with max_read_records controlling the maximum number of records to read.

make_builtin_decoder 🔒

make_decoder 🔒