feat: API/tool actuator tier — perform writes via API when available, GUI fallback#72
Merged
Conversation
Make structural (DOM/accessibility) evidence a first-class ACTION rung — the deterministic top of the resolution ladder — not just an identity signal. On structure-bearing backends the runtime re-finds the recorded target as a DOM/UIA element and acts on its center deterministically, falling back to the visual ladder (template/ocr/geometry/grounder) only where structure is absent (pixel-only substrates: RDP/Citrix/canvas). Two external reviews + the desktop benchmark converge here: UIA execution 21/21 vs compiled visual replay 6/21. Ladder: API → tool/MCP → [structural DOM/UIA] → template → template_global → ocr → geometry → grounder(VLM) → human. `structural` is rung 0, above `ocr`, so an irreversible step may act on it (strongest evidence). The visual rungs are unchanged — the fallback floor for pixel-only substrates. - ir: StructuralLocator (selector / role+name / UIA AutomationId) on Anchor.structural; StructuralHandle; "structural" added to Rung. - backend: optional StructuralActionBackend protocol (structural_locator_at + locate_structural). - resolver: structural rung first; falls through unchanged on miss/pixel-only. - playwright/windows backends: DOM (#id / role+name, with an occlusion hit-test) and UIA (AutomationId / role+name) locate. - recorder/compiler: capture the locator at record time; keep the visual anchor. - replayer: structural resolution flows through the SAME click path, so the identity gate + risk gate still fire; exempt from healing (deterministic locate ≠ stale template). New use_structural flag (default True) lets the visual-floor characterization suites exercise the pixel-only path. Availability measured in benchmark/structural_action (21/21 vs 6/21). Identity gate proven to still abort a sibling on a structurally-resolved point. Occlusion safe-halt preserved. New coverage in tests/test_structural_rung.py and tests/e2e/test_structural_action.py. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CKrVJJy5jWVCkXAqgUqtqZ
…o feat/api-actuator-tier # Conflicts: # openadapt_flow/runtime/replayer.py
… GUI as fallback The EXECUTE half of the capability ladder (the reviews' 'where a real API exists, GUI-driving it is the wrong tool'). When a step carries a reachable ApiBinding, actuate the write via the API deterministically, confirm it with the EffectVerifier (non-CONFIRMED -> HALT), and skip GUI actuation; otherwise fall through to the structural->visual ladder unchanged. Fail-safe: an attempted-but-unknown API outcome HALTS rather than risk a double-write. Additive (no binding -> replays as today). $0 / zero model calls. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CKrVJJy5jWVCkXAqgUqtqZ
# Conflicts: # openadapt_flow/ir.py # openadapt_flow/recorder.py # openadapt_flow/runtime/replayer.py
# Conflicts: # openadapt_flow/runtime/replayer.py
ae100d6 to
9fba22b
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
API/tool actuator tier — the EXECUTE half of the capability ladder
Two external reviews said it plainly: "where a real API exists, GUI-driving it is the wrong tool — use the API." openadapt-flow already can VERIFY effects (
EffectVerifier, merged) and RESOLVE targets structurally (structural rung, #69), but it still always PERFORMS the write through the GUI. This PR adds the top of the ladder: when a step has a real API/tool binding, perform the write via the API (deterministic,$0, no model), confirm it with theEffectVerifier, and skip GUI actuation — falling through to the structural→visual ladder when no binding exists.This is the RFC's
apiimplementation of aTransitionContract(docs/design/WORKFLOW_PROGRAM_IR.md§4): one semantic effect, backend-specific impl (api→dom_uia→vision_rdp).What's added
ir.ApiBinding(additive optional onStep): a declarative REST/JSON call —kind(rest/fhir/mcp/tool),method,url_template,body_template,query,headers,expected_status,timeout_s, and the expectedeffectsit produces.{param}placeholders substitute from the run's typed params. Shaped so a FHIR/MCP/tool binding fits the same model.runtime/actuators/ApiActuator: performs the call deterministically and classifies every attempt into a fail-safe outcome (no-double-write contract):UNAVAILABLE— request never sent (connection refused / DNS / connect-timeout, or a missing param) → nothing written → safe to fall through to the GUI ladder.ACTUATED— sent, 2xx → write performed → confirm via EffectVerifier, skip GUI.HALT— sent but outcome unknown (read-timeout after bytes went out, non-2xx, post-send error) → the write may have landed → HALT (never accept, never GUI-retry).The connect-phase/read-phase split is exact in
requests(ConnectTimeoutsubclassesConnectionError;ReadTimeoutdoes not), so catchingConnectionErrorbeforeTimeoutclassifies correctly.Wired as the API tier at the top of
_run_step: if a step has a binding and anApiActuatoris configured, actuate →EffectVerifierconfirms (non-CONFIRMED → HALT) → GUI resolve/act skipped. Otherwise falls through unchanged.Safety / gating preserved
EffectVerifier. Its "identity" is the explicit API parameter (stronger than a pixel band) — no identity gate is weakened.step.effectsnorbinding.effects) or no verifier configured.$0guarantee: zero model calls on the API path.Tests (
tests/test_replayer_api_actuator.py, all green)Against the real MockMed
fault_serverREST API +RestRecordVerifier:$0, 0 model calls, one record written with the run's param);UNAVAILABLE/ACTUATED/HALTclassification.Full suite: 1028 passed, 16 skipped (pre-existing environmental skips).
Ties to EffectVerifier + emit-mcp
The confirmation reuses the same
EffectVerifierpath that gates a GUI write (_verify_effectsgained an optional explicit-effects arg; the GUI call is unchanged).ApiBinding.kindincludesmcp/toolas the same shape — the complement toemit-mcp(which manufactures an MCP tool from a GUI demo): where the demo's target already exposes an API, the binding lets the runtime call it instead of replaying the GUI.Reconciliation touch-points (stacks on #69; retarget to
mainafter #69/#70 merge)Replayer edits are minimal and localized:
ir.py—ApiBindingclass +Step.api_binding+StepResult.actuation(all additive/optional) +ApiBinding.model_rebuild().replayer.py— one import (Effect);api_actuatorctor kwarg + assignment; a top-of-_run_stepbranch (_try_api_tier); the new_try_api_tiermethod;_verify_effectsgains an optionaleffects=arg; arung_counts["api"]accounting line inrun().runtime/actuators/package (no conflict surface).🤖 Generated with Claude Code
https://claude.ai/code/session_01CKrVJJy5jWVCkXAqgUqtqZ