Add US input-mass parity benchmark; archive the sparse-57k comparison#5
Open
MaxGhenis wants to merge 1 commit into
Open
Add US input-mass parity benchmark; archive the sparse-57k comparison#5MaxGhenis wants to merge 1 commit into
MaxGhenis wants to merge 1 commit into
Conversation
A calibrated release can hit its full target surface while carrying ~$0 in input bases the prior certified release populates (populace #278: the sparse-57k default zeroed IRA/HSA/pension-contribution/childcare inputs, so CDCC repeal scored $0.00B). This adds the benchmark area that makes that class of regression an auditable, archived verdict per release pair: - benchmarks/us/input-mass-parity/: explicit-path runner around populace.build.gates.input_mass_parity_gate (the release-blocking gate from populace PR #279), README with the promotion rule, and a manifest pinning the certified dense f0af251 (bundle 4.18.7) and sparse-57k (4.18.8) artifacts by HF revision and sha256. - archive/2026-07-02-sparse-57k-input-mass/: decision record for the run that quantified #278 — FAIL, 72 material engine-input columns lost (assets, QBID bases, take-up flags, tips, rent, veterans benefits, workers comp, hours, plus the four #278 bases); diagnostic-only, drove populace #278/#279. - pyproject.toml resolves the unpublished populace shards from the live repo as git-subdirectory sources; uv.lock pins the exact populace commit and engine version, and every result embeds input sha256s and the resolved policyengine-us version. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds the benchmark area that would have caught populace #278 before certification: a calibrated release can hit its full target surface while carrying ~$0 in input bases the prior certified release populates, silently zeroing every reform that touches them.
benchmarks/us/input-mass-parity/— explicit-path runner aroundpopulace.build.gates.input_mass_parity_gate(the release-blocking gate from populace PR #279), README with the promotion rule, manifest pinning the certified dense f0af251 (bundle 4.18.7) and sparse-57k (4.18.8) artifacts by HF revision + sha256.archive/2026-07-02-sparse-57k-input-mass/— decision record + scorecard for the run that quantified #278: FAIL, 72 material engine-input columns lost (the four #278 bases plus roth-IRA desired, net worth/stock/bank/bond assets, QBID bases, everytakes_up_*flag, tips, rent, veterans benefits, workers comp, pre-LSR hours). Diagnostic-only: the sparse release was already certified; the record drove #278/#279.pyproject.tomlresolves the unpublished populace shards as git-subdirectory sources;uv.lockpins the populace commit; every result embeds input sha256s and the resolved policyengine-us version (1.755.5 for the archived run).Verification
Ran end-to-end from a clean checkout:
uv run benchmarks/us/input-mass-parity/run_input_mass_parity.py --reference-h5 <dense f0af251> --candidate-h5 <sparse 57k> --flat-incumbent-h5 <local ECPS, diagnostic> --out archive/2026-07-02-sparse-57k-input-mass→ gate FAIL with 72 named failures, 180 columns totalled; outputs committed are exactly that run's artifacts.The next sparse rebuild should pass this benchmark against the dense f0af251 reference, with reviewed waivers for anything intentionally dropped.
🤖 Generated with Claude Code