feat(agents): detached (background) agent-tool runs — durable completion, progress & milestones

[threepointone]

Closes #1752.

TL;DR

Adds a first-class detached mode to runAgentTool so a parent agent can dispatch a sub-agent, keep working, and be notified exactly once when the child finishes — even across parent Durable Object eviction. On top of that, sub-agents can stream ephemeral progress and emit durable milestones that surface live in the parent's UI and (for chat agents) can narrate themselves back into the conversation. This is the framework-owned version of the ~200 lines of poll / fast-path / idempotency / race-handling glue @rwdaigle described in #1752, plus the progress/milestone tier the RFC laid out.

// Dispatch and keep going — returns immediately with { runId }.
const { runId } = await this.runAgentTool(ImportAgent, {
  input,
  detached: { onFinish: "onImportDone", maxBudgetMs: 30 * 60_000 }
});

// Fires once, durably, even if this DO was evicted in between.
async onImportDone(run: AgentToolRunInfo, result: AgentToolLifecycleResult) {
  if (result.status === "completed") this.foldInImport(result.summary);
}

For chat agents there's a one-liner that injects the result back into the conversation so the model reacts to it:

await this.runAgentTool(ImportAgent, { input, detached: { notify: true } });

And the child can report progress / milestones as it works:

// Inside the child sub-agent (e.g. from a tool's execute):
await this.reportProgress({ fraction: 0.6, phase: "deploying", message: "Generating menu page…" });
await this.reportProgress({ milestone: "preview-ready", data: { url } });

Status: this started as a proposal and is now fully fleshed out and implemented. Phases 1–3 (detached dispatch + durable completion + cancel + docs), Think notify, phase 4a (ephemeral progress) and phase 4b (durable milestones) all ship here. Only the awaitable join point (awaitAgentToolMilestone, phase 4c) remains deferred behind a design addendum.

1. Detached dispatch + durable, exactly-once completion

runAgentTool(Cls, { input, detached: true | { onFinish, maxBudgetMs, notify, onMilestones, noProgressBudgetMs } }) returns DetachedRunAgentToolResult ({ runId, agentType, status }) immediately, without awaiting. The full run lifecycle — run row in cf_agent_tool_runs, agent-tool-event broadcast, child recovery, onAgentToolStart / onAgentToolFinish, cost — fires regardless, exactly as on the awaited path. Detached runs deliberately inherit no AbortSignal (the child must outlive the spawning turn); cancel explicitly with cancelAgentTool(runId).

Delivery is a two-tier mechanism with a claim+lease ledger:

• Fast path — a ctx.waitUntil off the dispatch cuts latency while the parent isolate is warm.
• Durable backbone — a self-arming this.schedule callback (_cfDetachedReconcileTick, backoff cadence [5s, 15s, 30s, 120s]) that inspects the child to terminal, survives parent DO eviction, and re-arms on parent startup whenever outstanding detached runs exist. Arming is serialized so concurrent detached dispatches in one turn can't race to create multiple backbones.
• Claim+lease — terminal delivery is guarded by per-slot *_claimed_at / *_delivered_at columns claimed atomically via SQLite rowsWritten under a DETACHED_DELIVERY_LEASE_MS lease. The happy path delivers exactly once; a crash mid-delivery re-delivers (at-least-once), never zero. onFinish (named method, resolved by keyof this so it survives rehydration) and the global onAgentToolFinish both fire.

This resolves the two sharp edges from #1752:

• null is not proof a run is gone. The reconciler treats a null inspect as "not yet reconciled," never as failure, and detached runs are excluded from the await-style interrupted sealing — so a poll racing the child's first write can't manufacture a spurious "outcome unconfirmed."
• Give-up needs a separate idempotency key from success. Two independent ledger slots — a soft budget give-up (interrupted / budget-exceeded) and a later real completion — deliver as two distinct events. A premature give-up can never dedupe away the child's real summary.

Bounded. An absolute maxBudgetMs ceiling (default 24h, configurable via the detachedMaxBudgetMs static option) gives up — surfaced as interrupted with the budget-exceeded reason — and tears the child down so an abandoned run can't hold a concurrency slot forever.

2. Cancellation

cancelAgentTool(runId) cancels a detached (or awaited) run by id through the same guarded path, so a wired onFinish still fires once with status: "aborted", and the terminal agent-tool-event is always broadcast to connected clients (a cancelled run's UI settles immediately).

3. Chat notify (@cloudflare/think)

detached: { notify: true } (or { notify: { source } }) injects the completion into chat via submitMessages, idempotent per runId + terminal status, so the model reacts to the result without you wiring onFinish by hand. Override formatDetachedCompletion(run, result) to customize or suppress the text.

4. Progress (4a) + durable milestones (4b)

A sub-agent (awaited or detached) can report mid-run progress that rides its own turn stream and re-broadcasts to the parent's connected clients, surfacing on AgentToolRunState.progress / .milestones via useAgentToolEvents — a background-runs tray can render a live bar / phase / milestone chips without drilling in.

• reportProgress({ fraction?, message?, phase?, data? }, { persist? }) — ephemeral, latest-wins coalesced (fraction >= 1 always flushes), broadcast as a transient data-agent-progress part. The run id is resolved from the active turn — no threading. A no-op (with a dev warning) on the base Agent and outside an active run.

• reportProgress({ milestone, data }) — promotes a signal to durable: one row per milestone (cf_agent_tool_milestones / cf_ai_chat_agent_tool_milestones) with a monotonic per-run sequence, broadcast as a persisted data-agent-milestone part so drill-in replay and a rehydrated parent both see it.

• onProgress(run, progress) parent hook fires best-effort for both progress and milestones (the snapshot carries progress.milestone so consumers can branch).

• Latest-snapshot persistence + recovery inspect via progress_json / last_signal_at, surfaced through inspectAgentToolRun().progress / .milestones.

• Resetting no-progress budget for detached runs. Once a detached child reports at least one signal, the backbone gives up if it then goes silent for detachedNoProgressBudgetMs (default 1h; per-run detached: { noProgressBudgetMs }), surfaced as interrupted / reason no-progress. A child that never reports is bounded only by the absolute ceiling — we never give up on a run merely for being slow.

• Think detached: { onMilestones } convenience. When a configured milestone lands, Think surfaces an idempotent synthetic chat message (keyed per (runId, name)) before the run finishes — from both the warm tail and the cold backbone, collapsed to at-most-once. Two modes (the string[] shorthand defaults to "narrate"):

  • "narrate" (default) — a synthetic assistant message injected directly (no inference): a cheap status line that does not trigger a turn.
  • "react" — a user-role turn so the model responds (steer / start dependent work). Costs a model turn.

  detached: { onMilestones: ["preview-ready"] }                          // narrate (default)
  detached: { onMilestones: { names: ["needs-approval"], mode: "react" } }

  Override wording via formatDetachedMilestone(run, milestone). Synthetic notify/milestone messages carry metadata.source so clients can render them as an agent event rather than a human turn (the example does this).

Observability

New events: agent_tool:detached:delivery_failed (a wired callback threw; the slot stays open for retry) and agent_tool:detached:live_count_warning (edge-triggered when live detached runs cross a threshold — a leak smoke alarm, since detached runs hold a concurrency slot for their whole life).

Public API

• runAgentTool(Cls, { …, detached }) overload → DetachedRunAgentToolResult.
• DetachedAgentToolConfig: { onFinish?, maxBudgetMs?, noProgressBudgetMs?, notify?, onMilestones? }.
• cancelAgentTool(runId) — idempotent explicit cancellation.
• reportProgress(...) on chat agents; onProgress(run, progress) parent hook.
• Static options detachedMaxBudgetMs (24h) and detachedNoProgressBudgetMs (1h).
• New AgentToolInterruptedReason members "budget-exceeded" and "no-progress".
• @cloudflare/think: detached: { notify, onMilestones }; formatDetachedCompletion / formatDetachedMilestone.

Storage / migration

Schema bumped to v11. New columns on cf_agent_tool_runs (added via addColumnIfNotExists, so existing DOs migrate forward in place): detached, detached_on_finish, detached_max_budget_at, detached_on_milestones, progress_json, last_progress_at, and the four ledger columns finish_claimed_at / finish_delivered_at / give_up_claimed_at / give_up_delivered_at. New per-host milestone tables cf_agent_tool_milestones / cf_ai_chat_agent_tool_milestones. cf_agent_tool_runs is created lazily (not part of the constructor DDL snapshot), so no snapshot change.

Tests

• packages/agents/src/tests/agent-tool-detached.test.ts — ledger fires onFinish + global hook exactly once; dedupes fast-path vs backbone races; delivers give-up and a later real completion as two independent slots; never re-delivers a give-up.
• packages/agents/src/chat/__tests__/agent-tools.test.ts — emitter coalescing + milestone emit; client reducer projection of progress/milestones (deduped by sequence, monotonic snapshot).
• packages/think/src/tests/submissions.test.ts — detached notify nested-admission drain; milestone notify idempotency across re-delivery in both react and narrate modes.

Example

examples/agents-as-tools has a research_background tool (detached: { notify, onMilestones: ["sources-gathered"] }) that returns immediately, emits a milestone mid-run, posts the result back into the chat on completion, and a cancelBackground(runId) callable — so reviewers can try the end-to-end flow in a preview build. The client renders a background-runs tray with progress + milestone chips and renders synthetic messages as agent events.

Scope (what's deferred)

Only the RFC's awaitable join point — awaitAgentToolMilestone (phase 4c) — is intentionally deferred, gated behind a short design addendum + a real consumer. Everything else from the RFC ships here.

Validation

• pnpm run check — all projects typecheck + lint + format ✅
• agents · @cloudflare/think · @cloudflare/ai-chat · agents-as-tools tests ✅

Review starting points

• design/rfc-detached-agent-tools.md — full design + rationale + phasing.
• packages/agents/src/index.ts — runAgentTool detached branch, _parseDetachedOption, _deliverDetachedTerminal (claim+lease), _detachedFastPath, _cfDetachedReconcileTick (backbone + no-progress budget), cancelAgentTool, _maybeDeliverDetachedMilestone.
• packages/agents/src/chat/agent-tools.ts — AgentToolProgressEmitter, progress/milestone reducer projection.
• packages/think/src/think.ts — _cfDetachedNotifyFinish, _deliverDetachedMilestone (narrate/react), milestone persistence.

cc @rwdaigle — direct response to your write-up; would love your eyes on the shape.

Made with Cursor

---

[changeset-bot]

🦋 Changeset detected

Latest commit: c8fc9bc

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 3 packages

Name	Type
agents	Minor
@cloudflare/think	Minor
@cloudflare/ai-chat	Minor

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

[pkg-pr-new]

Open in StackBlitz

agents

npm i https://pkg.pr.new/agents@1758

@cloudflare/ai-chat

npm i https://pkg.pr.new/@cloudflare/ai-chat@1758

@cloudflare/codemode

npm i https://pkg.pr.new/@cloudflare/codemode@1758

create-think

npm i https://pkg.pr.new/create-think@1758

hono-agents

npm i https://pkg.pr.new/hono-agents@1758

@cloudflare/shell

npm i https://pkg.pr.new/@cloudflare/shell@1758

@cloudflare/think

npm i https://pkg.pr.new/@cloudflare/think@1758

@cloudflare/voice

npm i https://pkg.pr.new/@cloudflare/voice@1758

@cloudflare/worker-bundler

npm i https://pkg.pr.new/@cloudflare/worker-bundler@1758

commit: c8fc9bc

[rwdaigle]

Super small convenience comment:

metadata.source needs to be caller-controlled, not framework-fixed. The Think notify sugar stamps a hardcoded metadata.source on the injected completion message. Any consumer that already routes, filters, or hides synthetic messages by their source tag has its own established naming convention — forcing the framework's string means every such consumer must either migrate its conventions or fork the notify path. The framework shouldn't dictate this taxonomy; it should let the caller supply the source (e.g. a notify: { source } option, or honoring a value set in formatDetachedCompletion) so detached completions slot into each app's existing message-classification scheme rather than introducing a parallel one.

[devin-ai-integration]

Devin Review found 2 new potential issues.

⚠️ 1 issue in files not directly in the diff

⚠️ Warm-path milestone notifications silently fail because the run-row query omits the configuration column (packages/agents/src/index.ts:8668-8678)

The milestone configuration is never read on the low-latency warm path (_readAgentToolRun at packages/agents/src/index.ts:8668) because its SELECT omits detached_on_milestones, so _observeForwardedProgress → _maybeDeliverDetachedMilestone always sees undefined for the configuration and returns early without delivering.

Impact: Live milestone chat notifications (the detached: { onMilestones } convenience) never fire while the parent isolate is warm; they only arrive later via the cold backbone reconcile, which has its own complete SELECT.

Column omission in _readAgentToolRun vs _cfDetachedReconcileTick

The reconcile backbone query at packages/agents/src/index.ts:8431 selects all detached columns including detached_on_milestones, detached_no_progress_budget_ms, and last_progress_at. But the shared _readAgentToolRun query at packages/agents/src/index.ts:8668 only selects: detached, detached_on_finish, detached_notify_source, detached_max_budget_at, and the four ledger columns — omitting detached_on_milestones.

The warm-path flow:

1. Child emits a milestone → _forwardAgentToolStream calls _observeForwardedProgress(runId, body)
2. _observeForwardedProgress parses the milestone, calls this._readAgentToolRun(runId)
3. The returned row has detached_on_milestones as undefined (not selected)
4. _maybeDeliverDetachedMilestone evaluates row.detached_on_milestones ?? null → null → _parseAgentToolJson(null) → undefined → no configured names → returns without delivering

The TypeScript type marks the field as optional (detached_on_milestones?: string | null), so no compile error surfaces the omission.