Skip to content

Fix scheduler backpressure reporting and goal loop timeouts#7

Open
andrei-hasna wants to merge 3 commits into
feat/open-repos-multi-repo-loopsfrom
fix/scheduler-backpressure-timeouts
Open

Fix scheduler backpressure reporting and goal loop timeouts#7
andrei-hasna wants to merge 3 commits into
feat/open-repos-multi-repo-loopsfrom
fix/scheduler-backpressure-timeouts

Conversation

@andrei-hasna

@andrei-hasna andrei-hasna commented Jun 24, 2026

Copy link
Copy Markdown
Contributor

Summary

  • report due scheduler slots as backpressured when daemon concurrency is fully occupied, so a fresh daemon heartbeat no longer hides due work that could not be claimed
  • enforce executable loop target timeouts around goal-wrapped command targets and workflow step goals, returning timed_out instead of letting long Codewith goal runs hold slots indefinitely
  • keep timed-out and max-turn-limited goal runs retryable by resetting active nodes on timeout and reactivating retryable usageLimited goals on rerun
  • surface failed goal-node stdout/stderr excerpts in goal-loop --show-output, so Codewith profile/API failures no longer collapse to only node init failed: process exited with code 1
  • bound successful goal-node stdout/stderr before persisting goal evidence or feeding achievement validation, retaining head and tail excerpts to avoid megabyte-scale context fan-in
  • keep failed-node and goal output redacted by default unless the caller explicitly asks for --show-output

Incident Context

This addresses task 076dcc99 from the 2026-06-24 spark01 watchdog pass and follow-up incident task c7bd04ba-3957-430f-9364-f11aef5ae7a2. The daemon lease was fresh, but due short command loops were not claimed while all active run slots were occupied, and goal-wrapped Codewith command loops could stay active beyond their target timeout.

The follow-up incident showed fresh Codewith-backed goal nodes failing at the first node with only node init failed: process exited with code 1 in loop runs, while the child process failure reason was hidden. Direct bounded profile smokes reproduced usage-limit failures for account005/account009; this PR makes that child failure visible in explicit run output without exposing it in default views.

The 2026-06-24T14:28Z watchdog pass also showed context/max-turn pressure: disk-memory-pressure run f81b1ee38498 persisted about 1.36 MB of successful goal evidence before goal max turns exhausted, and loose-artifact-cleanup run d06e9d66cb8e exceeded the 128k model context limit. This PR now bounds successful node evidence before it is stored and replayed into achievement audits.

No live daemon restart, process kill, loop pause/resume, loop history/database deletion, or secret exposure was performed during triage or validation.

Adversarial Review

A reviewer flagged four issues in the first slice: backpressure counted the already-running loop, aborts during planning/validation could surface as generic failures, timeout used terminal cancelled goal state, and there was no real command-target abort regression. This revision reconciles those findings by excluding running/terminal/non-retryable/same-loop-overlap slots from the backpressure count, mapping timeout aborts to timed_out across planning/execution/validation, using retryable usageLimited state, and adding a real goal-wrapped command timeout test.

A follow-up reviewer found no blocking issues with the failed-node output patch. The residual risk was explicit --show-output exposure; this revision adds coverage that default loop-run formatting still redacts goal stdout/stderr unless show-output is requested.

A later adversarial review of the evidence-bounding patch found one blocking issue: marking max-turn exhaustion as blocked could strand retries with pending nodes. This was reconciled by keeping max-turn exhaustion retryable as usageLimited and adding a regression proving a second run resumes a pending dependent node. The same review noted tail-only capping could hide early proof, so excerpts now retain both head and tail output.

Validation

  • bun test src/lib/goal/runner.test.ts
  • bun test src/lib/goal/runner.test.ts src/lib/workflow-runner.test.ts src/lib/format.test.ts
  • bun run typecheck
  • git diff --check
  • bun test (111 tests across 17 files)
  • bun run build

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant