perf: parallel finite validation and scratch-based rank encode#279
Open
Nelson Spence (Fieldnote-Echo) wants to merge 1 commit into
Open
perf: parallel finite validation and scratch-based rank encode#279Nelson Spence (Fieldnote-Echo) wants to merge 1 commit into
Nelson Spence (Fieldnote-Echo) wants to merge 1 commit into
Conversation
assert_all_finite paid a full serial pass per add/search batch — measured ~0.1s per GiB, twice per ingest batch counting the caller layer. Scans of 1M+ floats now split across the rayon pool (4.4x measured). RankQuant::add's per-row closure allocated a fresh ranks Vec per vector inside the parallel loop; for_each_init now reuses a per-worker scratch via rank_transform_into. Measured on the 1.26M x 1024 corpus slice: encode-path attribution 0.097s serial scan -> 0.022s parallel; alloc churn removed from the hot loop.
Nelson Spence (Fieldnote-Echo)
requested review from
Navi Bot (project-navi-bot) and
Todd Baur (toadkicker)
as code owners
Member
Author
|
Codex (@codex) review |
|
You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard. |
1 similar comment
|
You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard. |
Codecov Report❌ Patch coverage is
📢 Thoughts on this report? Let us know! |
Member
Author
|
/agentic_review |
Code Review by Qodo🐞 Bugs (0) 📘 Rule violations (0) 📎 Requirement gaps (0)
Great, no issues found!Qodo reviewed your code and found no material issues that require review |
This was referenced Jul 4, 2026
project-navi-bot
approved these changes
Jul 4, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
assert_all_finitepaid a full serial pass per add/search batch — measured 0.097s per GiB. Inputs ≥1M floats now split the sweep across the rayon pool (0.097s → 0.022s measured on a real-corpus GiB slice); the panic message is unchanged.RankQuant::add's parallel closure allocated a freshranksVec per row;for_each_initnow reuses per-worker scratch viarank_transform_into.Measured (262,144 × 1024 real-corpus slice, best-of-3)
RankQuant::addSignBitmap::addOrdinalIndex::add_2d(downstream)Companion to ordinaldb
perf/parallel-input-validation(same fix at the caller layer, preserving exact first-offender panics). Independent of #277/#278 — branches off main, touches onlyutil.rs/quant.rs.Full gate: 250 tests + no-default-features + clippy -D warnings green.
Part of the 1M-row release train (encode lane).