docs: changelog perf entries + 0.6.0 un-patch checklist

CHANGELOG.md

@@ -9,6 +9,88 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 _No unreleased changes._
 
+## 0.6.0 - 2026-07-04
+
+### Performance
+
+- **Batched sign candidate generation now streams the corpus once per call.**
+  `SignBitmap::top_m_candidates_batched_serial_csr` previously looped the
+  single-query path, re-streaming the full sign bitmap per query (the
+  documented-naive first cut). The internals now scan the corpus once per call
+  in L2-sized doc blocks, score every query of the call against each hot block
+  in query tiles via the existing batched kernel, and select per-query top-m
+  with bounded `(hamming, doc_id)` min-collectors — bit-identical to a full
+  sort by construction, independent of processing order (the key *is* the
+  contract's sort key), pinned by an independent oracle suite
+  (`tests/tiled_candgen.rs`) across random, tie-heavy, duplicate-run, and edge
+  geometries. Per-query corpus traffic drops by the call's query count: at
+  1.26M rows × dim=1024, a 2048-query call reads the 161 MB sign sidecar once
+  instead of 2048 times. `top_m_candidates` routes through the same core
+  (dropping its per-call n-row Hamming materialisation) except at `nq=1`,
+  which keeps the dense partition path — the streamed core measured +50–90%
+  single-query time at small/medium `n` with `m` in the hundreds (bounded heap
+  `O(n log m)` vs `select_nth_unstable_by` `O(n)`), while the dense path is
+  parity-or-better at every measured size. The serial contract covers the
+  candidate scan and selection (no rayon there; callers own that
+  parallelism) — input finite-validation on large query buffers may
+  briefly use the global rayon pool (order-independent, deterministic).
+  `top_m_candidates_batched` (the internally-parallel convenience) is
+  unchanged by this work. Together with the collector worst-bound change below, measured
+  downstream in a two-stage retrieval stack at 1.26M × 1024: batched search
+  throughput 220 → 10.2k queries/s, results bit-identical.
+- **Candidate-collector accept test reduced to a cached worst-bound compare.**
+  Doc ids visit each per-query heap strictly ascending, so a candidate tying
+  the worst kept hamming always loses the `(hamming, doc_id)` tie-break — once
+  the collector is full, the accept test is exactly `hamming < worst kept
+  hamming`. That bound is now cached in a register-friendly `u32` (`u32::MAX`
+  while filling), skipping the heap peek + tuple compare on the ~99.8% reject
+  path. Bit-identical by construction; pinned by the tie-heavy and
+  duplicate-run oracle suites.
+- **Parallel finite-input validation and scratch-based rank encode.**
+  `assert_all_finite` paid a full serial pass per add/search batch — measured
+  ~0.1 s per GiB, twice per ingest batch counting the caller layer. Scans of
+  1M+ floats now split across the rayon pool (4.4× measured).
+  `RankQuant::add`'s per-row closure allocated a fresh ranks `Vec` per vector
+  inside the parallel loop; it now reuses a per-worker scratch via
+  `rank_transform_into`. Measured on a 1.26M × 1024 corpus slice: encode-path
+  validation attribution 0.097 s serial scan → 0.022 s parallel, with the
+  per-vector allocation churn removed from the hot loop.
+- **LUT + parallel constant-composition check on `RankQuant` load.**
+  `load_rankquant`'s forged-buffer defense histogrammed every packed code
+  serially — 1.29 billion shift/mask ops at 1.26M × 1024, ~1 s of the 1.27 s
+  verified open. A 4 KB per-byte bucket-count LUT replaces the per-code inner
+  loop and rows validate in parallel; `find_first` keeps the
+  lowest-offending-row error contract, with a scalar recheck producing the
+  identical message. The security property is unchanged: every row still
+  proves uniform composition before the index is usable. Measured verified
+  open at 1.26M × 1024: 1.27 s → 0.38 s.
+
+### Changed
+
+- **ordvec-manifest: derived artifact size bounds.** Verification now bounds
+  every artifact read by its manifest-declared `file_size_bytes` (the manifest
+  itself remains hard-capped at 1 MiB and SHA-256 pins content); manifest
+  creation bounds reads by the artifact's observed size. The flat
+  `ResourceLimits` byte caps (`max_auxiliary_artifact_bytes`,
+  `max_calibration_profile_bytes`, `max_encoder_distortion_profile_bytes`)
+  are now explicit opt-in ceilings and default to unbounded — previously the
+  64 MiB auxiliary default made legitimate large sign sidecars (>524,288 rows
+  at dim=1024) impossible to write with default options.
+- **ordvec-manifest: primary artifact reads are now bounded.** The primary
+  index artifact is hashed under its declared size (new
+  `artifact_file_too_large` reason code); previously this read was unbounded.
+  An artifact grown past its declaration now fails fast at the read bound
+  instead of surfacing as a digest mismatch after hashing the excess.
+- **ordvec-manifest: primary index artifact gains an opt-in ceiling.** New
+  `ResourceLimits::max_index_artifact_bytes` (default unbounded) mirrors the
+  auxiliary/profile ceilings; the create path also bounds the primary read by
+  its observed size. Note: a grown artifact now surfaces as
+  `*_file_too_large` (fail-fast) rather than `*_file_size_mismatch`, which
+  now indicates truncation only.
+- **ordvec-manifest: bounded hashing streams with constant memory.**
+  `sha256_file_bounded` no longer materialises the file in memory before
+  hashing.
+
 ## 0.5.0 - 2026-06-19
 
 ### Security

Cargo.toml

@@ -1,6 +1,6 @@
 [package]
 name = "ordvec"
-version = "0.5.0"
+version = "0.6.0"
 edition = "2021"
 rust-version = "1.89" # AVX-512 intrinsics stabilized in 1.89.0; also clears the 1.87 floor from u64::is_multiple_of
 description = "Training-free ordinal & sign quantization for vector retrieval"

README.md

@@ -38,9 +38,10 @@ append-friendly, and graph-optional.
 > trec-covid run below; the harness also supports nfcorpus and fiqa. ordvec wins
 > single-query latency against exact `flat` on the committed 171K-doc run and on
 > operability (no build, no tuning, append-only); in the committed default-method
-> threaded view, HNSW still wins highly-parallel batched serving. Larger-corpus
-> and alternate-encoder studies are active research, not public release claims
-> until their artifacts land in this repository.**
+> threaded view, HNSW still leads highly-parallel batched serving, though 0.6.0's
+> once-per-call corpus streaming narrowed that margin (see the threaded view
+> below). Larger-corpus and alternate-encoder studies are active research, not
+> public release claims until their artifacts land in this repository.**
 
 **Public evidence snapshot.** The load-bearing result in this README is narrower
 than the research backlog: Harrier-Q8 embeddings on public BEIR data, scored
@@ -60,8 +61,8 @@ and the gap widens over the committed subsampling sweep:
 ![ordvec speedup over exact search grows with corpus size](https://raw.githubusercontent.com/Project-Navi/ordvec/main/benchmarks/beir/figures/scaling_curve.png)
 
 - **~100× faster than exact `flat`, single query, at 171K docs.** Single-query
-  latency: exact `flat` 56 ms vs ordvec `Sign→rq2` **0.53 ms** — the gap over `flat`
-  grows with the corpus (it is ~5× at 1K docs).
+  latency: exact `flat` 52.4 ms vs ordvec `Sign→rq2` **0.52 ms (≈101×)** — the gap
+  over `flat` grows with the corpus (it is ~4.4× at 1K docs).
 - **8–16× smaller for the reported qrel rows.** The b=2 rank code is 256 B/vector
   (16× smaller than 4096 B floats), b=4 is 512 B (8×), and the reported two-stage
   `sign→rq2` row accounts for both stage-1 sign codes and the RankQuant reranker
@@ -70,8 +71,10 @@ and the gap widens over the committed subsampling sweep:
   followed by RankQuant b=2 rerank. At **nDCG@10 within bootstrap noise of exact**
   (on trec-covid the ordinal rows even edge ahead; see [Benchmarks](#benchmarks)).
 - **vs HNSW (the honest public scale story).** On the committed trec-covid run,
-  ordvec wins single-query latency while HNSW wins the highly-parallel threaded
-  view. That is the public comparison here. At larger corpora, graph or shard
+  ordvec wins single-query latency (≈3× at batch 1) while HNSW leads the
+  highly-parallel threaded view — by 1.6× over `sign→rq2` and 1.2× over
+  `bitmap→rq2` after 0.6.0's batched candidate generation (previously ≈2.3×).
+  That is the public comparison here. At larger corpora, graph or shard
   layers are the right comparison target; this README does not claim public
   million-scale HNSW crossover or GPU bandwidth numbers until the underlying run
   artifacts are committed.
@@ -209,7 +212,7 @@ Details in [`docs/RANK_MODES.md`](docs/RANK_MODES.md).
 
 ```toml
 [dependencies]
-ordvec = "0.5"
+ordvec = "0.6"
 
 # Or, to track unreleased `main`, use a git dependency instead:
 # ordvec = { git = "https://github.com/Project-Navi/ordvec" }
@@ -384,13 +387,13 @@ run; regenerate your own with `make benchmark-beir`.
 
 | Dataset | Method | Bytes/vec | nDCG@10 | Δ vs flat (95% CI) |
 |---|---|--:|--:|---|
-| scifact (5,183) | `flat` (exact) | 4096 | 0.7551 | (baseline) |
-| | `hnsw` M=32 | 4096 + graph | 0.7554 | +0.0003 * |
-| | **ordvec rq4** | **512** | **0.7549** | −0.0003 * |
-| | ordvec rq2 | 256 | 0.7471 | −0.0080 * |
-| | ordvec sign→rq2 | 384 | 0.7471 | −0.0080 * |
+| scifact (5,183) | `flat` (exact) | 4096 | 0.7559 | (baseline) |
+| | `hnsw` M=32 | 4096 + graph | 0.7573 | +0.0014 * |
+| | **ordvec rq4** | **512** | **0.7580** | +0.0021 * |
+| | ordvec rq2 | 256 | 0.7484 | −0.0075 * |
+| | ordvec sign→rq2 | 384 | 0.7484 | −0.0075 * |
 | trec-covid (171,332) | `flat` (exact) | 4096 | 0.7574 | (baseline) |
-| | `hnsw` M=32 | 4096 + graph | 0.7555 | −0.0019 * |
+| | `hnsw` M=32 | 4096 + graph | 0.7600 | +0.0026 * |
 | | ordvec rq2 | 256 | 0.7632 | +0.0057 * |
 | | **ordvec rq4** | **512** | **0.7636** | +0.0062 * |
 | | ordvec sign→rq2 | 384 | 0.7638 | +0.0064 * |
@@ -411,34 +414,38 @@ views (trec-covid, 171,332 docs, 1024-d):
 
 ![single-query latency bars](https://raw.githubusercontent.com/Project-Navi/ordvec/main/benchmarks/beir/figures/bars_single_thread.png)
 
-`flat` 56 ms → ordvec `sign→rq2` **0.53 ms (≈106×)**, `bitmap→rq2` 0.62 ms (≈91×),
-`hnsw` 1.5 ms (37×). The scaling curve [above](#benchmark-at-a-glance) is this
+`flat` 52.4 ms → ordvec `sign→rq2` **0.52 ms (≈101×)**, `bitmap→rq2` 0.58 ms (≈90×),
+`hnsw` 1.5 ms (≈34×). The scaling curve [above](#benchmark-at-a-glance) is this
 view swept over the committed subsamples — the speedup over `flat` grows across
 that public sweep.
 
 **2. Batched throughput (batch = 32, 1 thread)** — when many queries arrive at
-once, `flat`'s GEMM amortizes the corpus stream across the batch (56→4 ms),
-narrowing the gap: ordvec `sign→rq2`/`bitmap→rq2` stay ≈8–9.5× ahead.
+once, `flat`'s GEMM amortizes the corpus stream across the batch (52→3.8 ms).
+Since 0.6.0, ordvec's batched candidate generation amortizes the same way — the
+serial CSR path streams the corpus **once per call** instead of once per query
+(1.69× on the committed synthetic two-stage bench) — so `sign→rq2` 0.33 ms /
+`bitmap→rq2` 0.38 ms stay **≈10–12× ahead** of batched `flat`.
 
 **3. Many cores (batch = 32, 32 threads)** — everything parallelizes and the
 field compresses; `hnsw` threads best:
 
 ![threaded throughput bars](https://raw.githubusercontent.com/Project-Navi/ordvec/main/benchmarks/beir/figures/bars_threaded.png)
 
-`hnsw` 4.8× vs `flat`, ordvec `bitmap→rq2` 3.7×, `rq2` 2.5×, `sign→rq2` 2.1×.
+`hnsw` 4.9× vs `flat`, ordvec `bitmap→rq2` 4.0×, `sign→rq2` 3.1×, `rq2` 2.2×.
 This committed chart uses the default `sign-rq2` row, not the newer
 within-query-threaded `sign-rq2-threaded` probe row; regenerate public figures
 before using that probe for release claims. In this default-method view,
-**HNSW wins this regime** — by a hair on threaded throughput. The honest
+**HNSW still leads this regime** — 1.6× over `sign→rq2` (≈2.3× before 0.6.0's
+once-per-call corpus streaming) and 1.2× over `bitmap→rq2`. The honest
 ordvec-vs-HNSW tradeoff, all from this same run (trec-covid, 171,332 docs):
 
 | | HNSW M=32 | ordvec `sign→rq2` |
 |---|---|---|
-| threaded latency (32 threads, batch 32) | **0.23 ms** ✅ | 0.52 ms |
-| single-query latency (batch 1) | 1.52 ms | **0.53 ms** ✅ (~3×) |
+| threaded latency (32 threads, batch 32) | **0.20 ms** ✅ | 0.32 ms |
+| single-query latency (batch 1) | 1.52 ms | **0.52 ms** ✅ (~3×) |
 | index size / vector | 4096 B + graph | **256–384 B** ✅ (8–16× less) |
-| build time, 171K docs | **51.4 s** | **0.26 s** ✅ (training-free) |
-| nDCG@10 (trec-covid) | 0.7555 | **0.7638** ✅ |
+| build time, 171K docs | **47.1 s** | **0.21 s** ✅ (training-free) |
+| nDCG@10 (trec-covid) | 0.7600 | **0.7638** ✅ |
 
 So even where HNSW edges ahead on threaded latency, ordvec gets there with **no
 graph to build** (instant, training-free, and rebuilt for free when the corpus

RELEASING.md

@@ -173,6 +173,11 @@ the OIDC exchange (no risk of a bad publish; just a failed run).
      lockstep versions, MSRV/docs drift, registry metadata parity, Python
      classifier/URL parity, docs.rs feature policy, package contents, and
      release workflow invariants.
+   - **Downstream un-patch (one-time, 0.6.0):** OrdinalDB's workspace
+     `Cargo.toml` carries a `[patch.crates-io]` block pointing `ordvec` and
+     `ordvec-manifest` at this repo's `integration/full-stack` git branch.
+     When 0.6.0 publishes, that block must be removed so OrdinalDB consumes
+     the published crates.io releases instead of the pre-release git branch.
 4. Confirm CI is **green for current `main` HEAD**. `require-ci-green` checks
    `main` HEAD's SHA — which needs a **completed, successful** (not
    `cancelled`, not in-progress) run of `ci.yml`, `python.yml`, `fuzz.yml`,

THREAT_MODEL.md

@@ -1,6 +1,6 @@
 # Threat Model — `ordvec`
 
-> **Status:** v0.5.0 (pre-1.0), 2026-06-15. This is the maintained threat model
+> **Status:** v0.6.0 (pre-1.0), 2026-06-15. This is the maintained threat model
 > for the `ordvec` Rust crate, C ABI, Go wrapper, PyO3/maturin Python bindings,
 > and the `ordvec-manifest` sidecar verifier. It is reviewed when the
 > attack surface changes (new persistence formats, new `unsafe` kernels, new
@@ -397,6 +397,19 @@ enforce service-level quotas — by design (it is a library, not a server).
 batch size, `k`, request rate, and corpus size; a configurable `max_nq` /
 `max_k` at the binding level is a possible future convenience.
 
+**THREAT-QUERY-003 (P2): Artifact read bounds are derived, not flat.**
+Verification bounds every artifact read by its manifest-declared
+`file_size_bytes` (the manifest itself is hard-capped at 1 MiB before JSON
+parsing, and SHA-256 pins artifact content); manifest creation bounds reads
+by the artifact's observed size. Bounded hashing streams with constant
+memory, so a hostile manifest cannot cause unbounded memory growth — but it
+CAN still cause I/O and CPU proportional to the byte size it declares and
+actually supplies on disk. The flat `ResourceLimits` byte caps are opt-in
+ceilings (unbounded by default) for deployments that must bound worst-case
+verification time on attacker-supplied bundles. A `VerifiedLoadPlan` remains
+a verification snapshot, not a byte pin: bytes can change between
+verification and use by a local actor with write access (see scope).
+
 **THREAT-QUERY-002 (P3): Panic on contract violation in Rust server contexts.**
 Rust APIs fail fast on invalid contract input (non-finite floats, dimension /
 shape violations) via `assert!` / `expect`. In a Rust-native server an