Fix scheduler backpressure reporting and goal loop timeouts#7
Open
andrei-hasna wants to merge 3 commits into
Open
Fix scheduler backpressure reporting and goal loop timeouts#7andrei-hasna wants to merge 3 commits into
andrei-hasna wants to merge 3 commits into
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
backpressuredwhen daemon concurrency is fully occupied, so a fresh daemon heartbeat no longer hides due work that could not be claimedtimed_outinstead of letting long Codewith goal runs hold slots indefinitelyusageLimitedgoals on rerun--show-output, so Codewith profile/API failures no longer collapse to onlynode init failed: process exited with code 1--show-outputIncident Context
This addresses task
076dcc99from the 2026-06-24 spark01 watchdog pass and follow-up incident taskc7bd04ba-3957-430f-9364-f11aef5ae7a2. The daemon lease was fresh, but due short command loops were not claimed while all active run slots were occupied, and goal-wrapped Codewith command loops could stay active beyond their target timeout.The follow-up incident showed fresh Codewith-backed goal nodes failing at the first node with only
node init failed: process exited with code 1in loop runs, while the child process failure reason was hidden. Direct bounded profile smokes reproduced usage-limit failures for account005/account009; this PR makes that child failure visible in explicit run output without exposing it in default views.The 2026-06-24T14:28Z watchdog pass also showed context/max-turn pressure: disk-memory-pressure run
f81b1ee38498persisted about 1.36 MB of successful goal evidence beforegoal max turns exhausted, and loose-artifact-cleanup rund06e9d66cb8eexceeded the 128k model context limit. This PR now bounds successful node evidence before it is stored and replayed into achievement audits.No live daemon restart, process kill, loop pause/resume, loop history/database deletion, or secret exposure was performed during triage or validation.
Adversarial Review
A reviewer flagged four issues in the first slice: backpressure counted the already-running loop, aborts during planning/validation could surface as generic failures, timeout used terminal
cancelledgoal state, and there was no real command-target abort regression. This revision reconciles those findings by excluding running/terminal/non-retryable/same-loop-overlap slots from the backpressure count, mapping timeout aborts totimed_outacross planning/execution/validation, using retryableusageLimitedstate, and adding a real goal-wrapped command timeout test.A follow-up reviewer found no blocking issues with the failed-node output patch. The residual risk was explicit
--show-outputexposure; this revision adds coverage that default loop-run formatting still redacts goal stdout/stderr unless show-output is requested.A later adversarial review of the evidence-bounding patch found one blocking issue: marking max-turn exhaustion as
blockedcould strand retries with pending nodes. This was reconciled by keeping max-turn exhaustion retryable asusageLimitedand adding a regression proving a second run resumes a pending dependent node. The same review noted tail-only capping could hide early proof, so excerpts now retain both head and tail output.Validation
bun test src/lib/goal/runner.test.tsbun test src/lib/goal/runner.test.ts src/lib/workflow-runner.test.ts src/lib/format.test.tsbun run typecheckgit diff --checkbun test(111 tests across 17 files)bun run build