Add Complexity Tier rubric: per-story solver-tier recommendation (v2.18.0)#3
Merged
Conversation
…18.0) /case and /refine already gate scope/ambiguity via Budget-Solver Fit, but every story that passes it gets treated identically downstream — /solve only warns when the running model happens to be frontier-tier, never says which tier was actually cost-effective for that specific story. Complexity Tier is a new, separate rubric axis (judged after Budget-Solver Fit, never instead of it) that recommends the cheapest tier + effort combination likely to succeed: budget/medium/frontier x low/medium/high/max, with `medium` defined as a relative call (cheaper end of the planning roster, or strongest end of the budget roster) rather than a new Model Guard bucket. Escalation follows the signal: volume raises effort within a tier, subtlety/blast-radius (security, concurrency, judgment-heavy correctness) raises tier instead. Recorded two ways per story: a `## Complexity` section in the body, and a `solver-<tier>` bd label surfaced as a new column on /board. Two small follow-on fixes this surfaced: - /solve's existing frontier-cost warning fired on any frontier-tier run regardless of the story, contradicting a story that itself recommends frontier. Now checked once the story resolves, silent when the label already says solver-frontier. - /evaluate gains an optional post-merge calibration question (interactive flow only) recording whether the tier call matched reality, since nothing previously checked the rubric's accuracy over time. Reviewed by a Fable-tier model before implementation, which caught that the original draft's `medium` tier had no concrete referent and that its effort-before-tier escalation rule was backwards for the exact signals the rubric names — both reflected in the final version. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Complexity Tierrubric (shared/contract-rubrics.md, synced into/caseand/refine): a separate axis fromBudget-Solver Fitthat judges story difficulty (not scope/ambiguity) and recommends the cheapest solver tier + effort likely to succeed —budget/medium/frontier×low/medium/high/max.mediumis a relative call (cheaper end of the planning roster, or strongest end of the budget roster), not a new Model Guard bucket. Recorded per story as a## Complexitybody section plus asolver-<tier>bd label, surfaced as a new/boardcolumn./solve's existing frontier-cost warning is now story-aware: it no longer fires when the story's own label already recommendsfrontier(previously contradicted a deliberate, correct tier choice)./evaluategains an optional post-merge calibration question (interactive flow only) recording whether the tier recommendation matched reality, as a data point for judging the rubric's accuracy over time./case2.8.1→2.9.0,/refine1.7.1→1.8.0,/board1.0.0→1.1.0,/solve1.4.1→1.5.0,/evaluate1.8.0→1.9.0, plugin/marketplace 2.17.1→2.18.0 (all four manifests).This design was reviewed by a Fable-tier model before implementation, which caught that the original draft's
mediumtier had no concrete referent and that its effort-before-tier escalation heuristic was backwards for the exact signals the rubric names (subtlety/blast-radius needs a stronger tier, not more effort on a weaker one) — both are reflected in the final rubric.Test plan
plugins/case-solvers/tests/rubrics-sync.shpasses clean (no drift betweenshared/contract-rubrics.mdand the generated blocks in/case//refine)/case <description>on a frontier model for a mechanical story and a security/concurrency-touching one; confirm## Complexity+solver-<tier>label + recommendation surfaced in the completion report/boardrenders the new Solver column/refine <id>re-grades and swaps the label when a story proved harder than first judged/solve <id>on a frontier model: no cost warning against asolver-frontierstory, warning appears as before againstsolver-budget/unlabeled/evaluate <id>interactive approve flow asks the calibration question once;--approvefast path does not🤖 Generated with Claude Code