feat(agent/pr-reviewer): log diagnostic on fail-closed request-changes#65
Conversation
The recurring false-CHANGES_REQUESTED symptom (approve-looking body, event request-changes) is undiagnosable after the fact: the raw reviewBody the parser saw is lost once the Job pod is GC'd. When ParseVerdict fail-closes (empty / no verdict block / malformed JSON / unknown verdict), log the reason + reviewBody length + last 300 chars so the next recurrence is self-diagnosing from pod logs. Legitimate request-changes do not log. Pure helpers isFailClosedReason + lastChars added with table tests. No change to verdict/event mapping.
There was a problem hiding this comment.
Based on my thorough review of the diff, here is my assessment:
Files changed (5 files, +103/-0):
-
CHANGELOG.md (+4 lines) — Unreleased entry correctly describes the feature, purpose, and no behavior change to verdict/event mapping.
-
verdict.go (+29 lines) — Adds
isFailClosedReasonandlastChars:isFailClosedReasoncorrectly identifies all 4 fail-closed reasonsParseVerdictemits ("empty review text","no verdict block","malformed JSON:"*,"unknown verdict:"*) using exact equality for fixed strings andHasPrefixfor variable ones.lastCharsis rune-safe (converts to[]rune), handles all edge cases:n <= 0returns"", empty input returns"",len(s) <= nreturns full string, multi-byte chars are not split.
-
steps_checkout_execution.go (+16 lines) — The diagnostic
glog.Warningfis correctly gated byverdict.Verdict == VerdictRequestChanges && isFailClosedReason(verdict.Reason), ensuring it only fires for suspicious fail-closed cases. Logging does not alter control flow or verdict mapping. -
export_test.go (+10 lines) — Correctly exposes both helpers via
IsFailClosedReasonForTestandLastCharsForTestto thepkg_testpackage. -
verdict_diagnostics_test.go (+44 lines) — Comprehensive Ginkgo
DescribeTabletests:isFailClosedReason: 7 entries covering all 4 fail-closed + 3 non-fail-closed (including empty string)lastChars: 8 entries covering all edge cases + multi-byte rune non-split ("aé😀xy", 2,"xy")
The isFailClosedReason comment correctly documents the maintenance hazard (must stay in sync with ParseVerdict literals/prefixes) — this is an acceptable tradeoff since the alternative (exporting constants) would expose internal implementation details.
No issues found.
{
"verdict": "approve",
"summary": "This PR adds diagnostic logging for fail-closed request-changes verdicts. Two helper functions (isFailClosedReason, lastChars) are correctly implemented with rune-safe string handling. The diagnostic is appropriately gated and does not alter control flow. Tests are comprehensive with 15 table entries covering all edge cases.",
"comments": [],
"concerns_addressed": [
"correctness: isFailClosedReason correctly identifies all 4 ParseVerdict fail-closed reasons (verdict.go:178-183)",
"correctness: lastChars is rune-safe and handles all edge cases correctly (verdict.go:188-197)",
"correctness: glog.Warningf diagnostic in postAndRoute is gated and does not alter control flow or verdict mapping (steps_checkout_execution.go:311-318)",
"tests: verdict_diagnostics_test.go provides comprehensive table tests for both helpers including multi-byte character edge cases"
]
}
Why
The pr-reviewer intermittently posts GitHub
CHANGES_REQUESTEDwhile its review body is a cleanapprove(e.g. vault-ui PR #26, 2026-07-04 — admin-merged after owner OK). Root-causing it is blocked because the raw## ReviewtextParseVerdictactually saw is lost once the Job pod is GC'd — and on the posted body, currentParseVerdictcorrectly returnsapprove, so the failure lives in raw-output geometry we can't retrieve after the fact.This makes the failure legible: when it recurs, pod logs will name the reason and show the tail of the text the parser saw.
What
In
postAndRoute, afterParseVerdict, when the verdict isrequest-changesfrom a fail-closed reason (empty review text/no verdict block/malformed JSON:/unknown verdict:), emit:Legitimate (model-authored)
request-changesverdicts do not log — the gate isisFailClosedReason, so normal rejections stay quiet.isFailClosedReason+lastChars(rune-safe tail) inverdict.go, exposed viaexport_test.go, table-tested inverdict_diagnostics_test.go.Test
make precommitinagent/pr-reviewer— 13 packages green, 0 lint, 0 vulns.Follow-up
Once a recurrence is captured, the log pins whether it's trailing-content (
no verdict block), malformed JSON, or an unknown verdict — driving the actual parser fix. Tracked in[[Fix pr-reviewer False CHANGES_REQUESTED Regression]].