Skip to main content

Module legacy_fixed_len_byte_array

Module legacy_fixed_len_byte_array 

Source
Expand description

Temporary reader mitigation for arrow-rs #11261, fixed by #11262. These historical layouts are not additional Parquet encodings.

The ordinary decoder checks canonical PLAIN sizes before selecting a repair. A bounded prefix probe can rule out the historical layout, but a match never authorizes repair without an independent physical count. Repairs happen once per page; normal value decoding does not need to know the historical layout. Producer strings and embedded Arrow metadata never select a repair.

Canonical PLAIN has N * W bytes; the historical representation has exactly N * (W + 4) bytes with every little-endian length prefix equal to W. These sizes are disjoint for nonempty pages with the same independently established N. Without that count, the representations can overlap.

To retire the mitigation, remove this module, the compatibility hook in decoder::prepare_v1_fixed_len_byte_array, and the repair calls in the two normalization helpers there. Replace noncanonical PLAIN repair with a length error and reject FLBA/DELTA_LENGTH_BYTE_ARRAY instead of normalizing it. No changes to the column reader’s page setup, read/skip loops, byte-budget validation or canonical tests are required. Keep physical-value validation: it detects corruption independently of legacy support. Remove legacy fixture tests separately.

Re-exportsΒ§

use bytes::Bytes; πŸ”’
use super::decoder::PreparedFixedLenByteArrayPage; πŸ”’
use super::decoder::normalize_fixed_len_byte_array_data; πŸ”’
use crate::basic::Encoding; πŸ”’
use crate::data_type::Int32Type; πŸ”’
use crate::encodings::decoding::Decoder; πŸ”’
use crate::encodings::decoding::DeltaBitPackDecoder; πŸ”’
use crate::errors::ParquetError; πŸ”’
use crate::errors::Result; πŸ”’

FunctionsΒ§

decode_delta_length πŸ”’
Validate the BYTE_ARRAY-only delta-length representation and return its contiguous fixed-width values. The caller supplies the overflow-checked canonical payload size and passes the returned value section to PLAIN.
decode_length_prefixed_plain πŸ”’
Remove BYTE_ARRAY length prefixes from a noncanonical PLAIN FLBA section. The caller has checked expected_len = num_values * type_length for overflow and established that the input does not have that canonical length.
delta_length_value_count πŸ”’
Read the declared number of lengths, not an independent physical count. The caller must still check lengths, payload bytes and actual definition levels.
may_be_length_prefixed_plain πŸ”’
A negative result rules out the supported historical PLAIN layout. A positive result only requests an independent count: even every prefix matching could be an ordinary canonical payload. Bound the probe independently of page size.
prepare_v1_page πŸ”’
Handle nullable V1 pages that may need compatibility processing. The caller supplies a lazy, independent count; only ambiguous PLAIN or degenerate pages invoke it. Returns true if preparation is complete; false leaves page unchanged for the caller’s ordinary canonical path. Work in place so that ambiguous but canonical pages do not need additional buffer-reference clones.