Skip to main content

Module legacy_fixed_len_byte_array

Module legacy_fixed_len_byte_array 

Source
Expand description

Temporary reader mitigation for arrow-rs #11261, fixed by #11262. These historical layouts are not additional Parquet encodings.

The ordinary decoder checks canonical PLAIN sizes before selecting a repair. A bounded prefix probe can rule out the historical layout, but a match never authorizes repair without an independent physical count. Repairs happen once per page; normal value decoding does not need to know the historical layout. Producer strings and embedded Arrow metadata never select a repair.

Canonical PLAIN has N * W bytes; the historical representation has exactly N * (W + 4) bytes with every little-endian length prefix equal to W. These sizes are disjoint for nonempty pages with the same independently established N. Without that count, the representations can overlap.

To retire the mitigation, remove this module, the compatibility hook in decoder::prepare_v1_fixed_len_byte_array, and the repair calls in the two normalization helpers there. Replace noncanonical PLAIN repair with a length error and reject FLBA/DELTA_LENGTH_BYTE_ARRAY instead of normalizing it. No changes to the column readerโ€™s page setup, read/skip loops, byte-budget validation or canonical tests are required. Keep physical-value validation: it detects corruption independently of legacy support. Remove legacy fixture tests separately.

Functionsยง

decode_delta_length ๐Ÿ”’
Validate the BYTE_ARRAY-only delta-length representation and return its contiguous fixed-width values. The caller supplies the overflow-checked canonical payload size and passes the returned value section to PLAIN.
decode_length_prefixed_plain ๐Ÿ”’
Remove BYTE_ARRAY length prefixes from a noncanonical PLAIN FLBA section. The caller has checked expected_len = num_values * type_length for overflow and established that the input does not have that canonical length.
delta_length_value_count ๐Ÿ”’
Read the declared number of lengths, not an independent physical count. The caller must still check lengths, payload bytes and actual definition levels.
may_be_length_prefixed_plain ๐Ÿ”’
A negative result rules out the supported historical PLAIN layout. A positive result only requests an independent count: even every prefix matching could be an ordinary canonical payload. Bound the probe independently of page size.
prepare_v1_page ๐Ÿ”’
Handle nullable V1 pages that may need compatibility processing. The caller supplies a lazy, independent count; only ambiguous PLAIN or degenerate pages invoke it. Returns true if preparation is complete; false leaves page unchanged for the callerโ€™s ordinary canonical path. Work in place so that ambiguous but canonical pages do not need additional buffer-reference clones.