Proactive diagnostics: coverage board, certification panel, delta alerts, badges, CI (#101)#102
Open
MaxGhenis wants to merge 7 commits into
Open
Proactive diagnostics: coverage board, certification panel, delta alerts, badges, CI (#101)#102MaxGhenis wants to merge 7 commits into
MaxGhenis wants to merge 7 commits into
Conversation
…#101) New "Coverage" view (sidebar → Dataset accuracy) rendering the release's input-coverage picture: - Source coverage (us_source_coverage.json, published today): per hard-target source family, the package aliases it is required to cover, covers, or carries as a reviewed fiscal-refresh exclusion — each exclusion linking its tracking issue (#286 "cannot rot"). Validation-only families and source gaps too. - Input-column coverage (populace#369) and reform-coverage smoke (populace#368, the SSI-scores-$0 failure class): forward-compatible loaders that render a graceful "not published for this release" state until a build publishes input_coverage.json / reform_coverage_smoke.json — never fabricated data. issue-links.ts tokenizes inline issue references ("Issue #40", "populace#359", bare "#359", "owner/repo#N") into resolved GitHub URLs. GET /api/populace/coverage resolves the pointer once and serves all three artifacts. Bun tests cover the tokenizer and every parser against the real artifact shape. Refs #101 (buildable-now slice 1). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…rs (#101) New "Certification" view (sidebar → Releases) turning gate execution into a certification surface (populace#381): what the published evidence proves, not just what ran. - normalizeGate reads a gate into { outcome, enforced, evidence_sha, failures, reviewed_exclusions, stale_exclusions, summary }. It handles today's shape (passed:bool, metrics split between top-level and a `details` object) and populace#381's richer schema (explicit outcome passed|failed|skipped|waived + enforced + evidence sha) — the richer fields win when present, so #381 drops in without restructuring. - Side files (us_source_coverage / input_coverage / reform_coverage_smoke) are folded in as gates so the surface is exhaustive: published → pass/fail; unpublished → an honest "skipped — not published" row. - Reviewed-exclusion registers group every carried exclusion by gate, each linking its tracking issue (#286 cannot-rot); stale exclusions are tallied. GET /api/populace/certification serves the normalized surface. Verified against the live buildi manifest: 13 gates (10 manifest + 3 side), 11 passed / 2 skipped / 1 enforced, 20 reviewed exclusions across 2 registers. Bun tests cover current and #381 shapes plus the target_compilation derivation. Refs #101 (buildable-now slice 2). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…anner (#101) Turns the release diff into a push signal (epic item 3). - deltas.ts: one pure computeReleaseDelta over two releases → headline table (calibration loss, within-10%, records, targets), per-family reform-validation rows, coverage-shrink and gate-waived/skipped detection, each move judged against a per-metric-family band. Bands live in delta-bands.json (config, not code). loadReleaseDelta / loadLatestDelta assemble the report from HF. - scripts/populace-delta-alert.ts (bun): given two release ids (default latest vs previous), prints the delta table and posts to SLACK_WEBHOOK_POPULACE_US — the same var the populace publish CLI uses; a clear no-op log line when unset. Verified against two real releases (buildi vs the prior sparse refit): flags the +12.4pp within-10% jump, the -83% targets-included drop, the changed target surface, and a real 27→25 coverage shrink. - GET /api/populace/deltas/latest serves the same payload; the HF webhook now follows its release ping with the computed delta (best-effort). - DeltaBanner: an overview "since you last looked" banner (localStorage last-seen release id) with top movers and beyond-band flags; silent on first visit and once acknowledged. Bun tests cover band verdicts, reform/coverage/gate detection, non-comparable surfaces, and both renderers. Refs #101 (buildable-now slice 3). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
GET /api/populace/badge/{default-release,gates,within10} return shields.io
endpoint JSON ({ schemaVersion, label, message, color }) for READMEs and papers:
- default-release → the current build slug (e.g. "buildi")
- gates → passed/ran (skipped side-file gates excluded from the
denominator; red on any failure, green when all ran pass)
- within10 → the current release's within-10% share, color-graded
Served from latest.json + build_manifest.json (never the multi-MB diagnostics);
degrades to a valid lightgrey shield on error so a README badge never breaks.
Bun tests cover the slug parser and every color rule.
Refs #101 (buildable-now slice 4).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The repo had no CI (flagged in #97). ci.yml runs on PRs and pushes to main: bun install --frozen-lockfile, then typecheck (tsc --noEmit), bun test, and a next build — giving Pavel PR checks going forward. Verified locally: all three pass (build emits every new route/page). populace-delta-alert.yml is the scheduled safety net behind the HF webhook: a weekly run that posts to Slack only when the latest release moved beyond a band (--quiet-when-clean), plus on-demand dispatch for an arbitrary release pair. No-ops cleanly when the webhook secret is unset. Refs #101 (ops: minimal CI). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…101) Verification artifacts from a local next start against the live buildi release: headless screenshots of the Coverage and Certification views and the overview delta banner; the delta-script output over two real releases (webhook unset); the live badge endpoints; and the typecheck/test/build exit codes. Refs #101. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…e diff (#101) The "since you last looked" banner and the Slack delta alert link to /populace/compare?a=&b=. The compare view now reads those params to preselect the two releases (falling back to latest vs previous), so the deep-link lands on exactly the diff the alert is about. Refs #101. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Implements the buildable-now slice of the ratified epic #101 — turning the dashboard from a viewer into a watchdog. Everything reads live from the
policyengine/populace-usrelease (resolved throughlatest.json), builds on the #100 token layer (zero hardcoded hex, sentence case), and follows the existingloadReleaseUncached → buildCalibrationdata plumbing andcomponents/populace/view structure. Bound to the real artifact shapes in the currentbuildirelease; anything not yet published renders a graceful "not published for this release" state — never fabricated data.What's here
1. Reform-coverage board — new Coverage view (sidebar → Dataset accuracy). Renders
us_source_coverage.json(published today): 10 hard-target source families with required / covered / reviewed-excluded package aliases, each reviewed exclusion linking its tracking issue (#286 "cannot rot"), plus validation-only families and source gaps. The per-column input gate (populace#369) and the reform-coverage smoke (populace#368, the SSI-scores-$0 failure class) have forward-compatible loaders that light up the moment a build publishesinput_coverage.json/reform_coverage_smoke.json.2. Certification panel v1 — new Certification view (sidebar → Releases).
normalizeGateturns each gate into{ outcome, enforced, evidence_sha, failures, reviewed_exclusions, stale_exclusions, summary }, handling today's shape (metrics split between top-level and adetailsobject) and populace#381's richer schema (passed | failed | skipped | waived+ enforced + evidence sha) — the richer fields win when present, so #381 drops in without restructuring. The coverage/smoke side files are folded in as gates (published → pass/fail; unpublished → honest "skipped"). Reviewed-exclusion registers group every carried exclusion by gate.3. Release-delta alerts — one pure
computeReleaseDeltaover two releases → headline table (calibration loss, within-10%, records, targets), per-family reform-validation rows, coverage-shrink and gate-waived detection, each move judged against a per-metric-family band (delta-bands.json, config not code). It feeds three surfaces:scripts/populace-delta-alert.ts(bun) — default latest vs previous, prints the table, posts toSLACK_WEBHOOK_POPULACE_US(the var the populace publish CLI uses); clean no-op log when unset.GET /api/populace/deltas/latest— the same payload as a JSON feed; the HF webhook now also posts the computed delta on a new release.?a=&b=).4. Badge endpoints —
GET /api/populace/badge/{default-release,gates,within10}return shields.io endpoint JSON, served fromlatest.json+build_manifest.json(never the multi-MB diagnostics).5. Minimal CI —
.github/workflows/ci.yml(bun install --frozen-lockfile · typecheck · test · build on PRs and pushes) — the repo had none (flagged in #97). Plus a weeklypopulace-delta-alert.ymlsafety net behind the webhook.Verification
Local
next startagainst the livebuildirelease. Evidence committed indocs/proactive-evidence/:coverage-board.png,certification-panel.png,overview-delta-banner.png,delta-alert-output.txt(two real releases, webhook unset),badge-endpoints.txt.Delta over the last two real releases correctly flags the +12.4pp within-10% jump, the −83% targets-included drop (sparse vs dense), the changed target surface, and a real 27→25 source-coverage shrink. Badges resolve to
buildi · 11/11 · 88.9%.New bun tests: the issue-link tokenizer, every coverage/certification/deltas/badges parser (pinned to real artifact shapes), band verdicts, and forward-compat with the #381 schema.
Covers the buildable-now items of #101 (the attribution layer, promotion scorecards, and full BuildRunIdentity stay upstream-blocked as the epic notes).
Not merging — flagging for review.
🤖 Generated with Claude Code