Apache Arrow Go 18.9.0 Release


Published 16 Nov 2026
By The Apache Arrow PMC (pmc)

The Apache Arrow team is pleased to announce the v18.9.0 release of Apache Arrow Go. This minor release covers 152 commits from 27 distinct contributors.

Contributors

$ git shortlog -sn v18.8.0..v18.9.0
    45	Minh Vu
    37	Harrison Crosse
    31	Matt Topol
     6	singhpratech
     3	Sankalp Thakur
     2	Animesh Kumar
     2	Colton Loftus
     2	Edward Shen
     2	Samuele Resca
     2	Steve O'Brien
     2	Stochastic
     2	Willem Jan
     2	autoCAT
     2	matanper
     1	Arnaud
     1	Arnold Wakim
     1	Ayaan
     1	Jake Thomas
     1	Kyon
     1	Madan Kumar
     1	Nirajan Sah
     1	Ondřej Pavela
     1	Tobias Pütz
     1	hp
     1	peterolkhov
     1	prtkgaur
     1	ump45nose

Highlights

  • Go 1.26 is now the minimum supported Go version #1326

Arrow

Features

  • Added a scalar aggregate framework to the compute package, starting with the count and sum kernels #1336
  • Added the starts_with compute function, including string_view/binary_view inputs, IgnoreCase, and the Substrait starts_with mapping used by iceberg-go prefix row filters #1448, #1467, #1468
  • Added SortKey.FloatTotalOrder to sort floats by IEEE 754 totalOrder, distinguishing -0/+0 and signed NaNs #1430
  • Added Decimal32, Decimal64, StringView and BinaryView scalar types #1501, #1490
  • Added ipc.WithZstdLevel to choose the zstd compression level of IPC writers #1376
  • Added Reader.ContinueFrom to resume an IPC reader on a stream that arrives as several blobs, a pattern that stopped working in v18.8.0 #1346
  • The Flight SQL database/sql driver now accepts grpc+tcp://, grpc+tls:// and grpc+unix:// DSNs #1420
  • The JSON and CSV readers expose their input offset to help locate decoding errors #1342

Bug Fixes

  • is_null, is_not_null and not returned wrong masks for sliced arrays, dictionaries with null values, and null, union and run-end encoded arrays, so filters could silently keep or drop the wrong rows. is_null now also honors NullOptions.NanIsNull #1456, #1458, #1461, #1465, #1488
  • NewDatumWithoutOwning now returns a non-owning Datum; previously Release() freed the caller's memory #1377
  • and_kleene and or_kleene returned false instead of null when ExecCtx.ChunkSize was smaller than the input #1306
  • A cancelled CallFunction no longer returns a result with a nil error, and the default NumParallel follows the cgroup-aware GOMAXPROCS in containers #1410, #1384
  • Fixed generated assembly that clobbered the frame pointer or Go's reserved registers, breaking profilers and frame-pointer unwinders, in the amd64 compute kernels and the Parquet bit-unpacking and Bloom filter kernels #1432, #1300, #1379
  • Decimal rescaling no longer panics or returns wrong values, infinite decimal strings are rejected, and scales past MaxScale fail clearly #1307, #1409, #1513, #1514, #1515
  • The C Data interface exports extension arrays as their storage and imports zero-length fixed-size lists, such as those returned by DataFusion #1526, #1308
  • arrow.TypeEqual now compares LargeListType like ListType, so array.Concatenate no longer rejects identical large list types #1440
  • NewMappedFileReader no longer panics on dictionary fields, and Flight SQL ExecuteIngest sends the schema for empty inputs #1365, #1433
  • The JSON record reader no longer swallows an unexpected EOF on truncated input #1341

Performance Improvements

Performance remains a major theme: 51 of the merged pull requests are dedicated optimizations. Unless noted otherwise, the figures below are the PR-reported benchmarks on an Apple M1 Pro.

  • Bulk array comparisons: comparing an array without a validity bitmap against one with a materialized bitmap is up to 414x faster for booleans (8.8ms to 21.3us over 1M values) and 15.3x for int64, and all-valid list equality over 65,536 rows drops from 17.8ms to 0.197ms #1327, #1330, #1347
  • SIMD filtering now covers 8-, 16- and 64-bit types: 1M-element alternating int64 filters go from 3.97ms to 0.45ms (8.7x), int16 from 4.23ms to 225us, and 8-bit alternating masks run 22-24x faster. AVX2 take-index compaction adds up to 4.5x on AMD EPYC #1311, #1349, #1350, #1348
  • SortIndices radix-sorts single-chunk, null-free integer columns: 65,536 random int64 values sort in 1.31ms instead of 12.3ms #1391
  • Dictionary-heavy workloads: unique over 256 chunks with changing dictionaries is 36.6x faster (60.0ms to 1.64ms) with 98.5% fewer bytes allocated, concatenating chunks that share a dictionary drops from 82.4us to 3.7us, and builders reuse unchanged dictionaries, cutting the per-row cost of small-batch dictionary reads through pqarrow by up to 93% #1395, #1403, #1444
  • Bitmap utilities: AppendBools takes 88% less time (geometric mean), unaligned bitmap operations are up to 11.4x faster, and misaligned bitmap equality and unaligned transfers improve 6.5x and 5.4x (geometric means) #1402, #1437, #1414, #1416
  • IPC zstd compression: pooled decoders cut decompression time by 69-78% for buffers up to 256KiB per column, and pooled encoders take a writer-per-record workload from 246.4us to 4.5us and from about 2.2MiB to 7.1KiB allocated per op (Apple M2 Pro) #1331, #1429
  • Full-file CSV reads stream into builders: 1.69x faster with 71% fewer bytes allocated #1393

Parquet

Features

  • Added opt-in ALP (Adaptive Lossless floating-Point) encoding for FLOAT and DOUBLE columns, interoperable with Arrow C++. Writers enable it with WithALPEncoding; reading needs no option #1289
  • Added an optional parquet/encryption package with KMS support, so files encrypted by pyarrow's CryptoFactory (PKMT1 key material, single or double wrapping) can be read through KmsKeyRetriever #1478
  • Write geospatial statistics (bounding box and geometry types) for GEOMETRY column chunks, and null_count statistics for GEOMETRY and GEOGRAPHY #1504, #1533

Bug Fixes

  • Encryption interoperability: read encrypted files written by parquet-mr (Spark, Hive, Iceberg on parquet-java), preserve AES_GCM_CTR_V1 in plaintext footers, and return an error instead of panicking on columns without a key #1390, #1412, #1406
  • pqarrow wrote wrong data for sliced FixedSizeList columns and for FixedSizeList columns spanning more than one row group #1382
  • Encoders no longer carry stale state across pages: RLE boolean columns repeated earlier values on later pages, plain boolean and delta encoders wrote pre-Reset state, and all-null DELTA_BINARY_PACKED pages were padded with zeros #1539, #1545, #1546, #1400
  • Statistics: V2 data pages no longer report zero nulls without page statistics, byte array statistics written from Arrow count empty strings, and UpdateSpaced counts non-null values from the validity bitmap #1543, #1527, #1367
  • Fixed a memory leak in DELTA decoders, and dictionary-encoded boolean columns now return an error instead of panicking #1394, #1547

Performance Improvements

  • Reading dictionary-encoded columns allocates the dictionary up front again, fixing an allocation regression introduced in v18.8.0 #1473
  • Bloom filter hashing pools its scratch buffers: hashing 64 values drops from 5.52us to 311ns, and dictionary Bloom filter writes take up to 32% less time (Intel Core Ultra 7 155H) #1378
  • Level and boolean decoding: nullable boolean bitmap expansion takes 85% less time (geometric mean), 1-bit packed levels decode in 55-60% less time, all-true and all-false RLE booleans decode 66-98% faster (amd64), and narrow RLE literal levels encode 2.25-2.61x faster #1405, #1404, #1417, #1419
  • Dictionary encoding: indices are encoded in RLE batches (65,536 constant indices drop from 187.7us to 26.0us), and FLBA and Int96 dictionary insertion is 29-33% faster with up to 50% fewer allocations #1328, #1312
  • DELTA encodings: constant DELTA_BINARY_PACKED blocks decode 2.1x faster, DELTA_LENGTH_BYTE_ARRAY decodes 1.62-2.00x and discards 2.43-2.74x faster, DELTA_BYTE_ARRAY prefix matching is up to 4.7x faster (AMD EPYC), and its discard path drops from 65,535 allocations to none #1396, #1415, #1388, #1321
  • Reads: optional columns use dense decoding for batches without nulls (up to 45% faster per row on AMD EPYC), and serial reads build struct child readers without goroutines, 37-43% faster (Apple M4 Max) #1447, #1434

Changelog

Selected Changes

A selection of the user-facing changes in this release. See the full changelog link below for every change in the release, including automated dependency updates.

Arrow

  • feat(arrow/ipc): add Reader.ContinueFrom for streams split across blobs by @zeroshade in #1346
  • feat(arrow/ipc): add WithZstdLevel option for IPC writer by @Asthenia0412 in #1376
  • feat(arrow/flight/flightsql/driver): accept grpc+tcp, grpc+tls and grpc+unix schemes by @nirajan1111 in #1420
  • fix: expose internal offsets in json/csv readers by @Willem-J-an in #1342
  • feat(arrow/scalar): add StringView and BinaryView scalars by @hcrosse in #1490
  • feat(arrow/scalar): add Decimal32 and Decimal64 scalars by @hcrosse in #1501
  • fix(arrow): support zero-length fixed-size lists by @jakthom in #1308
  • fix(array/record): bldr stop ignore unexpected EOF by @Willem-J-an in #1341
  • fix(arrow/ipc): open mapped files with dictionary fields by @hpaliwal8 in #1365
  • fix(arrow): handle dense union slices and empty validation by @skyon0522-ai in #1385
  • fix(arrow/array): validate bufferless run-end encoded arrays by @zeroshade in #1421
  • fix(arrow/flight/flightsql): send schema for empty ingest by @arnoldwakim in #1433
  • fix(arrow): compare LargeListType like ListType in TypeEqual by @paveon in #1440
  • fix(arrow/bitutil): preserve overlapping bitmap transfers by @fallintoplace in #1389
  • fix(arrow/decimal): widen Decimal64 rescale to 64 bits by @winklemad in #1307
  • fix(arrow/decimal): handle native word widths in big integer conversions by @zeroshade in #1409
  • fix(arrow/decimal): correct rescaling results and report cast errors by @hcrosse in #1513
  • fix(arrow/decimal): reject infinite decimal strings and bound extreme exponents by @hcrosse in #1514
  • fix(arrow/decimal): fail clearly for Decimal128 and Decimal256 scales past MaxScale by @hcrosse in #1515
  • fix(arrow/cdata): export extension arrays as their storage by @hcrosse in #1526
  • fix(arrow/cdata): export an empty view sizes buffer as the zero region by @hcrosse in #1544
  • fix(arrow/array): validate extension arrays by their storage type by @hcrosse in #1487
  • fix(arrow/array): keep the storage dictionary in NewExtensionArrayWithStorage by @hcrosse in #1540
  • fix(arrow/csv): preserve braces in list values by @peterolkhov in #1499
  • fix(arrow/csv): preserve lists holding a single empty string by @zeroshade in #1505
  • fix(arrow/csv): keep list items equal to the null value distinct from nulls by @zeroshade in #1508

Compute

  • feat(arrow/compute): scalar aggregate framework with the count and sum kernels by @singhpratech in #1336
  • feat(arrow/compute): add IEEE 754 total-order float sorting option by @zeroshade in #1430
  • feat(arrow/compute): add starts_with kernel by @hcrosse in #1448
  • feat(arrow/compute): support string_view and binary_view in starts_with by @hcrosse in #1467
  • feat(arrow/compute): support IgnoreCase in starts_with by @hcrosse in #1468
  • fix(compute): detect temporal rounding overflow by @fallintoplace in #1127
  • fix(compute/exec): do not carry a cached null count across ArraySpan.SetSlice by @singhpratech in #1306
  • fix(arrow/compute): allow casting nullable struct fields to non-nullable when no nulls by @sankalpsthakur in #1363
  • fix(arrow/compute): fix NewDatumWithoutOwning ownership semantics by @Asthenia0412 in #1377
  • fix(arrow/compute): respect GOMAXPROCS for default NumParallel by @samueleresca in #1384
  • fix(arrow/compute): do not return a result from a cancelled call by @zeroshade in #1410
  • fix(arrow/compute): correct amd64 kernel assembly ABI by @zeroshade in #1432
  • fix(arrow/compute): honor the input offset in is_not_null by @hcrosse in #1456
  • fix(arrow/compute): register is_null, is_not_null and is_nan as unary functions by @hcrosse in #1457
  • fix(arrow/compute): honor the input offset in not by @hcrosse in #1458
  • fix(arrow/compute): treat null dictionary values as null in is_null and is_not_null by @hcrosse in #1461
  • fix(arrow/compute): handle null, union and run-end encoded arrays in is_null and is_not_null by @hcrosse in #1465
  • fix(arrow/compute): honor NullOptions.NanIsNull in is_null by @hcrosse in #1488
  • fix(arrow/compute/exprs): accept string and binary dictionary columns in field references by @hcrosse in #1449
  • fix(arrow/compute/exprs): fix offsets, nulls and ownership of nested field references by @zeroshade in #1474
  • fix(arrow/compute): use int16 kernels for casts to int16 by @hcrosse in #1503

Parquet

  • feat(parquet): add ALP encoding for FLOAT and DOUBLE columns by @prtkgaur in #1289
  • feat(parquet/encryption): KMS support for reading PKMT1 key material by @EdwardShen125 in #1478
  • feat(parquet/file): Write geospatial parquet row group statistics for GEOMETRY column chunks by @C-Loftus in #1504
  • fix(parquet/file): write null_count statistics for GEOMETRY and GEOGRAPHY column chunks by @C-Loftus in #1533
  • fix(parquet/utils): make unpack32 NEON profiler-safe by @zeroshade in #1300
  • fix(parquet/metadata): stop bloom filter asm clobbering the frame pointer by @zeroshade in #1379
  • fix(parquet/variant): prevent metadata offset cursor overflow by @matanper in #1290
  • fix(parquet/variant): return "" for an empty long-form string by @CaptainAni187 in #1353
  • fix(parquet/metadata): count non-null values from the validity bitmap in UpdateSpaced by @CaptainAni187 in #1367
  • fix(parquet/pqarrow): honor offset and null slots when writing fixed-size lists by @zeroshade in #1382
  • fix(parquet/file): read encrypted page headers written by parquet-mr by @sankalpsthakur in #1390
  • fix(parquet/internal/encoding): do not leak delta bit width buffers from decoders by @zeroshade in #1394
  • fix(parquet/internal/encoding): reset delta bit pack encoder state after flush by @zeroshade in #1400
  • fix(parquet): return an error instead of panicking on a hidden encrypted column by @ump45nose in #1406
  • fix(parquet): preserve CTR algorithm in plaintext footers by @autoCAT6 in #1412
  • fix(parquet/pqarrow): release partial readers when building a field reader fails by @elsteveogrande in #1435
  • fix(parquet): release writer buffers when a column chunk flush fails by @twuebi in #1506
  • fix(parquet/metadata): count empty strings in byte array statistics from Arrow by @hcrosse in #1527
  • fix(parquet/compress): read to the end of a Brotli stream that fills dst by @hcrosse in #1530
  • fix(parquet/internal/encoding): clear buffered RLE booleans after each flush by @hcrosse in #1539
  • fix(parquet/file): count V2 page num_nulls from the definition levels by @hcrosse in #1543
  • fix(parquet/internal/encoding): clear buffered RLE booleans on Reset by @hcrosse in #1545
  • fix(parquet/internal/encoding): clear encoder state on Reset by @hcrosse in #1546
  • fix(parquet/file): return an error for dictionary encoded boolean columns by @hcrosse in #1547

Performance

  • perf(arrow/array): compare optional validity bitmaps in bulk by @fallintoplace in #1327
  • perf(arrow/array): compare approximate list values by valid runs by @fallintoplace in #1330
  • perf(arrow/array): compare fixed-size binary values in bulk by @fallintoplace in #1347
  • perf(arrow/array): copy view buffers directly by @fallintoplace in #1324
  • perf(arrow/array): concatenate list offsets directly by @fallintoplace in #1323
  • perf(arrow/array): write bulk binary offsets and payloads directly by @fallintoplace in #1413
  • perf(arrow/array): reuse unchanged dictionaries across builder arrays by @hcrosse in #1444
  • perf(arrow/array): reuse dictionary mappings and handle empty chunks in Concatenate by @fallintoplace in #1403
  • perf(compute): vectorize mixed 64-bit filters by @fallintoplace in #1311
  • perf(compute): vectorize mixed 16-bit filters by @fallintoplace in #1349
  • perf(compute): vectorize mixed 8-bit filters by @fallintoplace in #1350
  • perf(compute): add AVX2 take-index compaction by @fallintoplace in #1348
  • perf(compute): reuse amd64 SIMD comparisons for numeric-to-bool casts by @fallintoplace in #1351
  • perf(arrow/compute): read flat take values directly by @fallintoplace in #1360
  • perf(arrow/compute): avoid getter wrappers in chunked binary take by @fallintoplace in #1359
  • perf(compute): radix-sort null-free integer indices by @fallintoplace in #1391
  • perf(arrow/compute): unify changing dictionaries incrementally by @fallintoplace in #1395
  • perf(compute): use typed memo insertion for binary hashes by @fallintoplace in #1315
  • perf(bitutil): pack boolean bitmap writes by byte by @fallintoplace in #1402
  • perf(bitutil): accelerate misaligned bitmap equality by @fallintoplace in #1414
  • perf(bitutil): bulk-transfer unaligned bitmap ranges by @fallintoplace in #1416
  • perf(bitutil): process unaligned bitmap operations by word by @fallintoplace in #1437
  • perf(arrow/csv): stream full-file CSV reads into builders by @fallintoplace in #1393
  • perf: zstd implement Pooling + DecodeAll with no new reader. by @samueleresca in #1331
  • perf(arrow/ipc): reuse zstd encoders across writers by @vdpoora in #1429
  • perf(parquet/metadata): pool bloom filter hash scratch buffers by @zeroshade in #1378
  • perf(parquet): use typed memo insertion for fixed-width dictionaries by @fallintoplace in #1312
  • perf(parquet): encode dictionary indices in RLE batches by @fallintoplace in #1328
  • perf(parquet): reuse DELTA_BYTE_ARRAY discard storage by @fallintoplace in #1321
  • perf(parquet): reuse DELTA decoder scratch by @fallintoplace in #1322
  • perf(parquet): speed up delta byte array prefix matching by @fallintoplace in #1388
  • perf(parquet): fast-path constant delta bitpacked blocks by @fallintoplace in #1396
  • perf(parquet): batch DELTA_LENGTH byte-array decoding and discard by @fallintoplace in #1415
  • perf(parquet): decode narrow packed levels directly by @fallintoplace in #1404
  • perf(parquet): expand nullable boolean bitmaps by byte by @fallintoplace in #1405
  • perf(parquet): decode RLE booleans directly into bool slices by @fallintoplace in #1417
  • perf(parquet): pack narrow RLE literal levels eight at a time by @fallintoplace in #1419
  • perf(parquet/file): use dense decoding for batches without nulls by @hcrosse in #1447
  • perf(parquet/pqarrow): build struct child readers serially when Parallel is false by @elsteveogrande in #1434
  • perf(parquet/internal/encoding): preallocate dictionaries bounded by page data by @zeroshade in #1473

New Contributors

  • @matanper made their first contribution in #1290
  • @singhpratech made their first contribution in #1303
  • @jakthom made their first contribution in #1308
  • @CaptainAni187 made their first contribution in #1353
  • @samueleresca made their first contribution in #1331
  • @hpaliwal8 made their first contribution in #1365
  • @sankalpsthakur made their first contribution in #1363
  • @Asthenia0412 made their first contribution in #1376
  • @skyon0522-ai made their first contribution in #1385
  • @nirajan1111 made their first contribution in #1420
  • @autoCAT6 made their first contribution in #1412
  • @vdpoora made their first contribution in #1429
  • @elsteveogrande made their first contribution in #1435
  • @EdwardShen125 made their first contribution in #1439
  • @AynSiddiqui made their first contribution in #1477
  • @peterolkhov made their first contribution in #1499
  • @prtkgaur made their first contribution in #1289

Full Changelog: https://github.com/apache/arrow-go/compare/v18.8.0...v18.9.0