Skip to content

feat: API/tool actuator tier — perform writes via API when available, GUI fallback#72

Merged
abrichr merged 5 commits into
mainfrom
feat/api-actuator-tier
Jul 13, 2026
Merged

feat: API/tool actuator tier — perform writes via API when available, GUI fallback#72
abrichr merged 5 commits into
mainfrom
feat/api-actuator-tier

Conversation

@abrichr

@abrichr abrichr commented Jul 13, 2026

Copy link
Copy Markdown
Member

API/tool actuator tier — the EXECUTE half of the capability ladder

Two external reviews said it plainly: "where a real API exists, GUI-driving it is the wrong tool — use the API." openadapt-flow already can VERIFY effects (EffectVerifier, merged) and RESOLVE targets structurally (structural rung, #69), but it still always PERFORMS the write through the GUI. This PR adds the top of the ladder: when a step has a real API/tool binding, perform the write via the API (deterministic, $0, no model), confirm it with the EffectVerifier, and skip GUI actuation — falling through to the structural→visual ladder when no binding exists.

This is the RFC's api implementation of a TransitionContract (docs/design/WORKFLOW_PROGRAM_IR.md §4): one semantic effect, backend-specific impl (apidom_uiavision_rdp).

What's added

  • ir.ApiBinding (additive optional on Step): a declarative REST/JSON call — kind (rest/fhir/mcp/tool), method, url_template, body_template, query, headers, expected_status, timeout_s, and the expected effects it produces. {param} placeholders substitute from the run's typed params. Shaped so a FHIR/MCP/tool binding fits the same model.

  • runtime/actuators/ApiActuator: performs the call deterministically and classifies every attempt into a fail-safe outcome (no-double-write contract):

    • UNAVAILABLE — request never sent (connection refused / DNS / connect-timeout, or a missing param) → nothing written → safe to fall through to the GUI ladder.
    • ACTUATED — sent, 2xx → write performed → confirm via EffectVerifier, skip GUI.
    • HALT — sent but outcome unknown (read-timeout after bytes went out, non-2xx, post-send error) → the write may have landed → HALT (never accept, never GUI-retry).

    The connect-phase/read-phase split is exact in requests (ConnectTimeout subclasses ConnectionError; ReadTimeout does not), so catching ConnectionError before Timeout classifies correctly.

  • Wired as the API tier at the top of _run_step: if a step has a binding and an ApiActuator is configured, actuate → EffectVerifier confirms (non-CONFIRMED → HALT) → GUI resolve/act skipped. Otherwise falls through unchanged.

Safety / gating preserved

  • An API write still faces the EffectVerifier. Its "identity" is the explicit API parameter (stronger than a pixel band) — no identity gate is weakened.
  • Fail-safe HALTs before any request when the write couldn't be confirmed: no effect contract (neither step.effects nor binding.effects) or no verifier configured.
  • Additive: a step with no binding — or a binding with no actuator configured — behaves exactly as today (GUI actuation). The API tier is an optimization whose safe fallback is the GUI, never a gate that blocks a runnable step.
  • $0 guarantee: zero model calls on the API path.

Tests (tests/test_replayer_api_actuator.py, all green)

Against the real MockMed fault_server REST API + RestRecordVerifier:

  • binding actuates → EffectVerifier CONFIRMS → GUI skipped ($0, 0 model calls, one record written with the run's param);
  • unreachable API falls through to the GUI cleanly — write happens exactly once, no double-write;
  • REFUTED effect after an API write HALTS (record, not screen, is the oracle);
  • attempted-but-rejected (non-2xx) HALTS, never GUI-retries;
  • fail-safe HALTs (binding with no effect; binding with no verifier);
  • back-compat: binding present + no actuator → GUI unchanged; no-binding bundle replays byte-identically;
  • unit tests for the actuator's UNAVAILABLE/ACTUATED/HALT classification.

Full suite: 1028 passed, 16 skipped (pre-existing environmental skips).

Ties to EffectVerifier + emit-mcp

The confirmation reuses the same EffectVerifier path that gates a GUI write (_verify_effects gained an optional explicit-effects arg; the GUI call is unchanged). ApiBinding.kind includes mcp/tool as the same shape — the complement to emit-mcp (which manufactures an MCP tool from a GUI demo): where the demo's target already exposes an API, the binding lets the runtime call it instead of replaying the GUI.

Reconciliation touch-points (stacks on #69; retarget to main after #69/#70 merge)

Replayer edits are minimal and localized:

  • ir.pyApiBinding class + Step.api_binding + StepResult.actuation (all additive/optional) + ApiBinding.model_rebuild().
  • replayer.py — one import (Effect); api_actuator ctor kwarg + assignment; a top-of-_run_step branch (_try_api_tier); the new _try_api_tier method; _verify_effects gains an optional effects= arg; a rung_counts["api"] accounting line in run().
  • new runtime/actuators/ package (no conflict surface).

🤖 Generated with Claude Code

https://claude.ai/code/session_01CKrVJJy5jWVCkXAqgUqtqZ

abrichr and others added 3 commits July 13, 2026 10:58
Make structural (DOM/accessibility) evidence a first-class ACTION rung — the
deterministic top of the resolution ladder — not just an identity signal. On
structure-bearing backends the runtime re-finds the recorded target as a
DOM/UIA element and acts on its center deterministically, falling back to the
visual ladder (template/ocr/geometry/grounder) only where structure is absent
(pixel-only substrates: RDP/Citrix/canvas). Two external reviews + the desktop
benchmark converge here: UIA execution 21/21 vs compiled visual replay 6/21.

Ladder: API → tool/MCP → [structural DOM/UIA] → template → template_global →
ocr → geometry → grounder(VLM) → human. `structural` is rung 0, above `ocr`,
so an irreversible step may act on it (strongest evidence). The visual rungs
are unchanged — the fallback floor for pixel-only substrates.

- ir: StructuralLocator (selector / role+name / UIA AutomationId) on
  Anchor.structural; StructuralHandle; "structural" added to Rung.
- backend: optional StructuralActionBackend protocol (structural_locator_at +
  locate_structural).
- resolver: structural rung first; falls through unchanged on miss/pixel-only.
- playwright/windows backends: DOM (#id / role+name, with an occlusion
  hit-test) and UIA (AutomationId / role+name) locate.
- recorder/compiler: capture the locator at record time; keep the visual anchor.
- replayer: structural resolution flows through the SAME click path, so the
  identity gate + risk gate still fire; exempt from healing (deterministic
  locate ≠ stale template). New use_structural flag (default True) lets the
  visual-floor characterization suites exercise the pixel-only path.

Availability measured in benchmark/structural_action (21/21 vs 6/21). Identity
gate proven to still abort a sibling on a structurally-resolved point. Occlusion
safe-halt preserved. New coverage in tests/test_structural_rung.py and
tests/e2e/test_structural_action.py.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CKrVJJy5jWVCkXAqgUqtqZ
…o feat/api-actuator-tier

# Conflicts:
#	openadapt_flow/runtime/replayer.py
… GUI as fallback

The EXECUTE half of the capability ladder (the reviews' 'where a real API
exists, GUI-driving it is the wrong tool'). When a step carries a reachable
ApiBinding, actuate the write via the API deterministically, confirm it with
the EffectVerifier (non-CONFIRMED -> HALT), and skip GUI actuation; otherwise
fall through to the structural->visual ladder unchanged. Fail-safe: an
attempted-but-unknown API outcome HALTS rather than risk a double-write.
Additive (no binding -> replays as today). $0 / zero model calls.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CKrVJJy5jWVCkXAqgUqtqZ
@abrichr abrichr changed the base branch from main to feat/structural-action-rung July 13, 2026 16:11
@abrichr abrichr changed the base branch from feat/structural-action-rung to main July 13, 2026 16:28
abrichr added 2 commits July 13, 2026 13:42
# Conflicts:
#	openadapt_flow/ir.py
#	openadapt_flow/recorder.py
#	openadapt_flow/runtime/replayer.py
# Conflicts:
#	openadapt_flow/runtime/replayer.py
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant