Azure ADLS Gen2 backend — 3 bugs found: app role LocalFileSystem, concurrent writer 409, compactor manifest format-version mismatch

[imtiazqa]

Built coldfront-duckdb15:pg18 locally on Apple Silicon Mac (arm64), transferred the image to a Rocky Linux 10 arm64 VM via docker save | gzip | scp | docker load, then ran make compactor && ci/topo/vanilla.sh --pg 18 --mode tiered --backend azure --compose docker-compose.matrix-azure.yml.

Before running set these environment vraiables in the session:-

export COLDFRONT_AZURE_ACCOUNT="..."
export COLDFRONT_AZURE_FILESYSTEM="..."
export COLDFRONT_AZURE_KEY="..."
export COLDFRONT_AZURE_CONNECTION_STRING="DefaultEndpointsProtocol=https;AccountName=${COLDFRONT_AZURE_ACCOUNT};AccountKey=${COLDFRONT_AZURE_KEY};EndpointSuffix=core.windows.net"

Result: 123 passed / 10 failed

Bug 1 — Non-superuser app role cold I/O fails on Azure (LocalFileSystem disabled)

File: extension/coldfront/coldfront--0.1.sql — ensure_attached() line 180

ensure_attached() is SECURITY DEFINER and pre-loads the iceberg extension inside the elevated context but never calls LOAD azure. On Azure, DuckDB auto-loads the azure extension on the first query — at that point the session is already running as the non-superuser japp, so pg_duckdb has LocalFileSystem disabled → extension file load fails. Works fine on S3/SeaweedFS because httpfs loads as a side-effect of the iceberg ATTACH with no further LocalFileSystem call.

ERROR: Permission Error: File system LocalFileSystem has been disabled by configuration
Failing: app role reads tiered hot+cold — expected '290', got 'SET' and app role cold write landed — expected '1', got '0'

Fix: detect storage type from coldfront.storage_secret inside ensure_attached() and call PERFORM duckdb.raw_query('LOAD azure') before returning, within the SECURITY DEFINER context.

Bug 2 — Concurrent mixed-tier writers get Conflict_409 on Azure (advisory lock + deferred POST race)

File: extension/coldfront/coldfront--0.1.sql — _exec_iceberg_with_claim() line 1964

The vanilla single-node path takes pg_advisory_xact_lock then calls duckdb.raw_query(). pg_duckdb's Iceberg commit POST is deferred to after the PostgreSQL transaction ends — which is also when the advisory lock is released. On SeaweedFS (local, sub-millisecond) the deferred POST fires before the next writer gets in. On Azure ADLS (network round-trip), the POST is slow enough that multiple writers release their locks and queue deferred POSTs simultaneously at Lakekeeper → all race on the same snapshot → Conflict_409. Same race surfaces as archiver errored during race window (phase 3 attempt 2, 409).

Failed to commit Iceberg transaction: Conflict_409, CatalogCommitConflicts
expected: 9099356468329532207, found: 562916686884392705
Failing: no concurrent mixed-tier writer errored — expected '0', got '12', 4 concurrent writers all landed — expected '8', got '2', archiver errored during race window

Bug 3 — Compactor fails on all backends (manifest format-version patch incomplete)

File: docker/iceberg-manifest-list-format-version-v15.patch

iceberg-manifest-list-format-version-v15.patch adds format-version to the manifest-list Avro file header metadata. However, iceberg-go (cmd/compactor) also validates the per-entry format-version field inside the manifest list records. The patch fixes the header but leaves the per-entry field at v1 — iceberg-go checks both and rejects on the mismatch. This is not Azure-specific; it fails on all backends. It was hidden because ci/topo/vanilla.sh only runs make build (archiver + partitioner) and never builds the compactor — stories 6d and 6e silently failed with No such file or directory in all prior runs. Running make compactor before the journey exposes the underlying patch bug.

compactor: plan files for default.events:
manifest file's 'format-version' metadata indicates version 2,
but entry from manifest list indicates version 1
Failing: compactor --dry-run found nothing to compact (6d), expire --dry-run shows nothing to expire (6e)

Also: ci/topo/vanilla.sh should run make compactor before the journey step so stories 6d/6e are never silently skipped again.

[imtiazqa]

Built coldfront-duckdb15:pg18 locally on Apple Silicon Mac (arm64) — Rocky Linux 10 arm64 VM cannot build it directly because Dockerfile.duckdb15-base hardcodes FROM quay.io/pypa/manylinux_2_28_x86_64 and bare Docker CE on Linux has no QEMU emulation (Mac Docker Desktop provides it automatically). Transferred the image to the Rocky Linux 10 arm64 VM via docker save | gzip | scp | docker load, then ran make compactor && ci/topo/vanilla.sh --pg 18 --mode tiered --backend azure --compose docker-compose.matrix-azure.yml.

[vyruss]

Thanks for the deep Azure pass. 2 of these are already fixed: Bug 1 (the non-superuser LOAD azure is now verified working end-to-end on real ADLS). The compactor CI gap where vanilla.sh never built bin/compactor is also fixed, so 6d/6e don't skip silently anymore.

I was a bit confused with Bugs 2 and 3, because I couldn't repro. Because of the build method, I suspect the cause is which iceberg extension duckdb has actually loaded and not the cold-write code. With autoload_known_extensions=on, if the patched iceberg/avro/azure aren't the ones loaded, DuckDB autoloads extensions from the repo. iceberg_bakery_patch=on asserts the patch and doesn't verify it. Stock iceberg lacks both patches: the bakery commit-refresh (without it, iceberg_async_parquet=on causes stale-parent CatalogCommitConflicts) and the manifest-list format-version patch (Bug 3).

How to check what extensions are loaded, in a session that's used the cold tier:

SELECT * FROM duckdb.query($$ SELECT extension_name, loaded, install_mode
FROM duckdb_extensions() WHERE extension_name IN ('iceberg','avro','azure') $$);

install_mode = REPOSITORY means it autoloaded the stock extension.

With the latest tag, the base should build natively on arm64 (no Mac, no QEMU) and prebuild the patched extensions. You can try building that way (docker/Dockerfile.duckdb15-base, DUCKDB_1.5_PATCHED.md), confirm the query shows the patched iceberg, and retry.

[imtiazqa]

ColdFront Azure ADLS Gen2 — Updated Findings (arm64 native build)
Environment: Rocky Linux 10 arm64 · commit 90bf842 · PG18 tiered mode · Azure ADLS Gen2
Command: ci/topo/vanilla.sh --pg 18 --mode tiered --backend azure --compose docker-compose.matrix-azure.yml
Result: 123 passed / 10 failed

Confirmed Fixed

Fix | Commit | Verified -- | -- | -- Bug 1: app role LocalFileSystem disabled on Azure | 90bf842 | ✅ PASS: app role reads tiered hot+cold + PASS: app role cold write landed Compactor not built before journey | 90bf842 | ✅ make compactor now runs automatically in vanilla.sh arm64 Dockerfile — native build on Rocky arm64 | 59d70da | ✅ Builds without QEMU or Mac

Remaining Issue — DuckDB Version Mismatch (arm64 only)
Root cause of all 10 remaining failures.

DuckDB binary version: v1.5.4 ← pg_duckdb PR #1025 resolves v1.5.4 on arm64
COLDFRONT_DUCKDB_VERSION env: v1.5.3 ← entrypoint places patched extensions here
install_mode (iceberg/avro/azure): REPOSITORY ← DuckDB downloaded stock extensions
Patched extensions sit in .../v1.5.3/linux_arm64/. DuckDB looks in .../v1.5.4/linux_arm64/, finds nothing, downloads stock from repository.

Patched extensions sit in .../v1.5.3/linux_arm64/. DuckDB looks in .../v1.5.4/linux_arm64/, finds nothing, downloads stock from repository.

# | Failures | Story -- | -- | -- ×2 | Compactor manifest format-version mismatch | 6d, 6e ×5 | Concurrent mixed-tier writers 409 | 6b ×1 | Archiver race window 409 | 9 ×2 | NEW Tiered cold writers 409 (was PASS on amd64) | 9b

Fix needed: Pin pg_duckdb PR #1025 to DuckDB v1.5.3 on arm64, or bump COLDFRONT_DUCKDB_VERSION + OVERRIDE_GIT_DESCRIBE to v1.5.4 and rebuild patched duckdb-iceberg against v1.5.4.

[vyruss]

That's it. The pinned PR 1025 was merged today into pg_duckdb, so I've now gone and pinned duckdb 1.5.4 tag officially and set COLDFRONT_DUCKDB_VERSION/OVERRIDE_GIT_DESCRIBE to v1.5.4. I've rebuilt the patched duckdb-iceberg against it and the full matrix passes.