Publish incumbent-comparison scorecards (schema + pointer + validator)

[PavelMakarchuk]

Fixes #3.

What

Gives the US incumbent comparison a machine-readable, validated scorecard artifact so downstream consumers (the calibration-diagnostics dashboard) can read the populace-vs-enhanced-CPS head-to-head without re-running the harness.

• archive/us/populace-us-2024-9f1260b-20260611/scorecard.json — the first published scorecard, status: archived (reconstructed from that release's sound_ecps_replacement_comparison.json, since the comparison was correctly dropped from live populace in populace#37). Carries the promotion metrics (full/holdout loss, unweighted MSRE), per-target win/loss/tie (populace 1,040 vs eCPS 2,613 of 3,704), per-family loss breakdown, and top movers.
• benchmarks/us/incumbent-comparison/latest.json — pointer to the current scorecard (scorecard_path + candidate_release_id), so consumers don't hard-code a path (mirrors the populace#9 latest.json pattern).
• scorecard.schema.json — the published contract (JSON Schema draft-07).
• tools/validate_scorecard.py — stdlib-only validator: required keys, promotion-metric types, and the consistency the schema can't express — win counts sum to the target count, and candidate_beats_baseline agrees with the loss values. Verified it rejects both kinds of inconsistency.
• CI (.github/workflows/validate-scorecards.yml) runs the validator on every PR — the repo's first gate.

Boundary

Respects the repo's rules: no candidate is built or discovered from a working dir; the scorecard is a committed result keyed to the certified pinned-production-ecps-2024 incumbent, and archived status is explicit so it isn't mistaken for a fresh promotion-valid run.

Validation

$ python tools/validate_scorecard.py
OK: 2 file(s) valid (1 scorecard(s), 1 pointer(s)).

Consumer

The dashboard's incumbent-comparison view reads this artifact (a companion PR points its live fetch at latest.json, falling back to its committed snapshot until this merges).

🤖 Generated with Claude Code

[MaxGhenis]

Solid, and the validator does what the description claims — I ran it against tampered copies (agent-assisted) and it correctly rejects a broken win-count partition, a candidate_beats_baseline that disagrees with the losses, wrong types, a missing required key, and a bad schema_version. The archived scorecard is internally consistent (family n_targets sum to 3,704; per-family win/tie totals reconcile to 1,040/2,613/51) and conforms to its own draft-07 schema. Provenance and status: archived are honest about the reconstruction, and the third-party surface is clean — only census/IRS/JCT target names and our own enhanced_cps baseline.

Two things worth doing before this becomes the enforced published contract:

1. CI doesn't actually enforce the schema. The validator is stdlib-only and hand-rolls a subset of checks; it never validates documents against scorecard.schema.json, so anything the schema expresses but the validator doesn't — summary field types, n_targets >= 1, family/top-mover item shapes — can drift or land malformed. The required-key tuples in the validator also duplicate the schema's required lists by hand. Either add an optional jsonschema pass (skip gracefully if unavailable) or gate on it in CI; #5 adds pyproject/uv.lock, so uv run --with jsonschema is cheap after that.

2. Gate vocabulary mismatch. Issue #3 and the incumbent-comparison README describe export/support/lineage gates, but the scorecard carries a single ecps_refit_recovery gate. Fine for an archived reconstruction, but reconcile the vocabulary (or note it in provenance) so a dashboard doesn't build UI expecting gates that aren't there.

Smaller: candidate_beats_baseline is only checked against full loss, though the promotion rule is full+holdout+MSRE (harmless here — populace wins all three); the pointer validator checks candidate_release_id but not incumbent_manifest; the schema $id isn't a resolvable URL. All non-blocking.

Consumer note, not a code change: populace wins aggregate loss decisively (0.228 vs 1.405) but loses most per-target head-to-heads (1,040 vs 2,613) — consistent, since it wins the high-weight targets, but any surface showing win counts should show the loss context alongside so it doesn't read as populace losing.

Still marked draft, and #5 has moved ahead of it (both touch README, so whoever merges second needs a trivial rebase). Happy to approve once it's marked ready — none of the above blocks.