Expand description
Temporary reader mitigation for arrow-rs #11261, fixed by #11262. These historical layouts are not additional Parquet encodings.
The ordinary decoder checks canonical PLAIN sizes before selecting a repair. A bounded prefix probe can rule out the historical layout, but a match never authorizes repair without an independent physical count. Repairs happen once per page; normal value decoding does not need to know the historical layout. Producer strings and embedded Arrow metadata never select a repair.
Canonical PLAIN has N * W bytes; the historical representation has exactly
N * (W + 4) bytes with every little-endian length prefix equal to W. These
sizes are disjoint for nonempty pages with the same independently established
N. Without that count, the representations can overlap.
To retire the mitigation, remove this module, the compatibility hook in
decoder::prepare_v1_fixed_len_byte_array, and the repair calls in the two
normalization helpers there. Replace noncanonical PLAIN repair with a length
error and reject FLBA/DELTA_LENGTH_BYTE_ARRAY instead of normalizing it.
No changes to the column readerโs page setup, read/skip loops, byte-budget
validation or canonical tests are required. Keep physical-value validation:
it detects corruption independently of legacy support. Remove legacy fixture
tests separately.
Functionsยง
- decode_
delta_ ๐length - Validate the BYTE_ARRAY-only delta-length representation and return its contiguous fixed-width values. The caller supplies the overflow-checked canonical payload size and passes the returned value section to PLAIN.
- decode_
length_ ๐prefixed_ plain - Remove BYTE_ARRAY length prefixes from a noncanonical PLAIN FLBA section.
The caller has checked
expected_len = num_values * type_lengthfor overflow and established that the input does not have that canonical length. - delta_
length_ ๐value_ count - Read the declared number of lengths, not an independent physical count. The caller must still check lengths, payload bytes and actual definition levels.
- may_
be_ ๐length_ prefixed_ plain - A negative result rules out the supported historical PLAIN layout. A positive result only requests an independent count: even every prefix matching could be an ordinary canonical payload. Bound the probe independently of page size.
- prepare_
v1_ ๐page - Handle nullable V1 pages that may need compatibility processing. The caller
supplies a lazy, independent count; only ambiguous PLAIN or degenerate pages
invoke it. Returns
trueif preparation is complete;falseleavespageunchanged for the callerโs ordinary canonical path. Work in place so that ambiguous but canonical pages do not need additional buffer-reference clones.