feat: structural (DOM/UIA) action rung — vision-first, not vision-only

DESIGN.md

@@ -5,10 +5,16 @@ deterministic, vision-anchored script → replay it locally with a resolution
 ladder → when the UI drifts, heal the script (as a reviewable diff) instead of
 re-reasoning every run.
 
-The runtime is **vision-only**: it consumes PNG bytes and emits clicks/keys at
-pixel coordinates through the `Backend` protocol (`openadapt_flow/backend.py`).
-The reference backend is Playwright-driven (headless-capable, CI-friendly,
-permission-free); native OS / RDP backends are future adapters.
+The runtime is **vision-first, not vision-only**: it always CAN operate a pure
+surface — consuming PNG bytes and emitting clicks/keys at pixel coordinates
+through the `Backend` protocol (`openadapt_flow/backend.py`) — but where a
+backend owns a structured layer (a browser DOM, a native UIA/AX tree) the
+resolution ladder's TOP rung re-finds the recorded target as an ELEMENT and
+acts on it deterministically (`StructuralActionBackend`, see Resolution ladder).
+Structure is preferred where present; the visual ladder is the fallback floor
+for pixel-only substrates (RDP/Citrix/canvas). The reference backend is
+Playwright-driven (headless-capable, CI-friendly, permission-free); native OS /
+RDP backends are future adapters.
 
 ## Frozen contracts (do not change without updating this doc)
 
@@ -149,7 +155,27 @@ class Replayer:
 Parameters not supplied in `params` fall back to the recorded example/default
 values in `workflow.params`, so a bundle replays without any explicit params.
 
-Resolution ladder per step with an anchor (record rung + confidence + ms):
+Resolution ladder per step with an anchor (record rung + confidence + ms).
+The full capability hierarchy is API → tool/MCP → DOM/UIA → geometry → OCR →
+template → VLM → human; API and tool/MCP are future placeholders, so the
+implemented rungs are:
+
+0. `structural` — DETERMINISTIC (additive change, structural-rung spike). When
+   the live backend implements `StructuralActionBackend` (a browser DOM, a
+   native UIA/AX tree) AND the anchor carries a `StructuralLocator` the compiler
+   captured (a stable DOM selector / role+name, or a UIA AutomationId /
+   role+name), the runtime re-finds the recorded target as an ELEMENT via
+   `backend.locate_structural(locator)` and acts on its center — no pixel
+   matching. Tried FIRST; a successful locate short-circuits the visual rungs.
+   Pixel-only substrates (RDP/Citrix/canvas) and failed/ambiguous locates fall
+   through to the visual rungs below UNCHANGED. This is the thesis refinement
+   from "vision-only" to "deterministic compiled automation with visual
+   FALLBACK" (desktop benchmark: UIA 21/21 vs compiled visual replay 6/21 under
+   drift; reproduced by `benchmark/structural_action/`). The resolved point
+   flows through the SAME click path, so the pre-click identity gate and the
+   irreversible risk gate still fire on it — structure makes identity STRONGER
+   (an exact element), it never bypasses it. Rungs 1–5 remain the visual
+   FALLBACK floor and are unchanged:
 1. `template` — find_template within anchor.region padded by search_pad
 2. `template_global` — find_template full frame; for UNLABELED anchors
    (no ocr_text) the match is rejected when every locatable landmark places

benchmark/structural_action/STRUCTURAL_ACTION.md

@@ -0,0 +1,52 @@
+# Structural action rung — availability under render drift
+
+Reproduces, on a real rendered DOM with the real resolution ladder, the desktop
+benchmark that reframed the thesis from **"vision-only"** to **"deterministic
+compiled automation with visual FALLBACK"**: structural (DOM/UIA) execution
+scored **21/21** while compiled *visual* replay scored **6/21** under render
+drift.
+
+## What it measures
+
+`structural_action_probe.py` (logic in
+`openadapt_flow/validation/structural_action.py`) renders a dense surface of
+`n` actionable targets, each with a stable DOM id (`#open-pK`) — what a browser
+recorder captures for the structural rung — plus a recorded visual anchor
+(template crop + OCR label). It then RE-RENDERS with drift: a deterministic
+subset is left untouched (steps whose surface did not change), the rest are
+MOVED, RELABELLED (`Open`→`View`) and RE-THEMED (dark, serif). The DOM id never
+changes. For each target it asks, on the drifted surface:
+
+- **structural** — `backend.locate_structural(locator)`: does it land inside the
+  CORRECT element's box? (Deterministic, pixel-independent.)
+- **visual** — `resolve(anchor, drift_png, vision, structural=None)`: does the
+  template/OCR/geometry ladder resolve to a point inside the CORRECT box? A
+  drifted target's recorded crop no longer matches at its old position and
+  either fails or matches a look-alike sibling — both are failures.
+
+Numbers are MEASURED, never hardcoded.
+
+## Result (`structural_action.json`, n=21)
+
+```
+structural (DOM) rung : 21/21 acted correctly
+visual ladder only    :  6/21 acted correctly
+under drift (15 targets): structural 15/15 vs visual 0/15
+```
+
+Structural resolves every target whose id is still in the DOM; the visual ladder
+resolves only the non-drifted minority. The structural point flows through the
+SAME click path, so the pre-click identity gate and the irreversible risk gate
+still fire on it — structure makes identity STRONGER (an exact element), it
+never bypasses it.
+
+## Run it
+
+```
+python benchmark/structural_action/structural_action_probe.py 21
+```
+
+The in-suite assertion of the same win (structural resolves all, beats visual
+under drift, identity gate intact) lives in `tests/test_structural_rung.py`; the
+full record→compile→replay default-on path is in
+`tests/e2e/test_structural_action.py`.

benchmark/structural_action/structural_action.json

@@ -0,0 +1,177 @@
+{
+  "n": 21,
+  "structural_ok": 21,
+  "visual_ok": 6,
+  "structural_ratio": "21/21",
+  "visual_ratio": "6/21",
+  "targets": [
+    {
+      "id": "open-p0",
+      "drifted": false,
+      "structural_ok": true,
+      "structural_rung": true,
+      "visual_ok": true,
+      "visual_rung": "template"
+    },
+    {
+      "id": "open-p1",
+      "drifted": true,
+      "structural_ok": true,
+      "structural_rung": true,
+      "visual_ok": false,
+      "visual_rung": "template"
+    },
+    {
+      "id": "open-p2",
+      "drifted": true,
+      "structural_ok": true,
+      "structural_rung": true,
+      "visual_ok": false,
+      "visual_rung": "template"
+    },
+    {
+      "id": "open-p3",
+      "drifted": true,
+      "structural_ok": true,
+      "structural_rung": true,
+      "visual_ok": false,
+      "visual_rung": "template"
+    },
+    {
+      "id": "open-p4",
+      "drifted": false,
+      "structural_ok": true,
+      "structural_rung": true,
+      "visual_ok": true,
+      "visual_rung": "template"
+    },
+    {
+      "id": "open-p5",
+      "drifted": true,
+      "structural_ok": true,
+      "structural_rung": true,
+      "visual_ok": false,
+      "visual_rung": "template"
+    },
+    {
+      "id": "open-p6",
+      "drifted": true,
+      "structural_ok": true,
+      "structural_rung": true,
+      "visual_ok": false,
+      "visual_rung": "template"
+    },
+    {
+      "id": "open-p7",
+      "drifted": true,
+      "structural_ok": true,
+      "structural_rung": true,
+      "visual_ok": false,
+      "visual_rung": "template"
+    },
+    {
+      "id": "open-p8",
+      "drifted": false,
+      "structural_ok": true,
+      "structural_rung": true,
+      "visual_ok": true,
+      "visual_rung": "template"
+    },
+    {
+      "id": "open-p9",
+      "drifted": true,
+      "structural_ok": true,
+      "structural_rung": true,
+      "visual_ok": false,
+      "visual_rung": "template"
+    },
+    {
+      "id": "open-p10",
+      "drifted": true,
+      "structural_ok": true,
+      "structural_rung": true,
+      "visual_ok": false,
+      "visual_rung": "template"
+    },
+    {
+      "id": "open-p11",
+      "drifted": true,
+      "structural_ok": true,
+      "structural_rung": true,
+      "visual_ok": false,
+      "visual_rung": "template"
+    },
+    {
+      "id": "open-p12",
+      "drifted": false,
+      "structural_ok": true,
+      "structural_rung": true,
+      "visual_ok": true,
+      "visual_rung": "template"
+    },
+    {
+      "id": "open-p13",
+      "drifted": true,
+      "structural_ok": true,
+      "structural_rung": true,
+      "visual_ok": false,
+      "visual_rung": "template"
+    },
+    {
+      "id": "open-p14",
+      "drifted": true,
+      "structural_ok": true,
+      "structural_rung": true,
+      "visual_ok": false,
+      "visual_rung": "template"
+    },
+    {
+      "id": "open-p15",
+      "drifted": true,
+      "structural_ok": true,
+      "structural_rung": true,
+      "visual_ok": false,
+      "visual_rung": "template"
+    },
+    {
+      "id": "open-p16",
+      "drifted": false,
+      "structural_ok": true,
+      "structural_rung": true,
+      "visual_ok": true,
+      "visual_rung": "template"
+    },
+    {
+      "id": "open-p17",
+      "drifted": true,
+      "structural_ok": true,
+      "structural_rung": true,
+      "visual_ok": false,
+      "visual_rung": "template"
+    },
+    {
+      "id": "open-p18",
+      "drifted": true,
+      "structural_ok": true,
+      "structural_rung": true,
+      "visual_ok": false,
+      "visual_rung": "template"
+    },
+    {
+      "id": "open-p19",
+      "drifted": true,
+      "structural_ok": true,
+      "structural_rung": true,
+      "visual_ok": false,
+      "visual_rung": "template"
+    },
+    {
+      "id": "open-p20",
+      "drifted": false,
+      "structural_ok": true,
+      "structural_rung": true,
+      "visual_ok": true,
+      "visual_rung": "template"
+    }
+  ]
+}

benchmark/structural_action/structural_action_probe.py

@@ -0,0 +1,39 @@
+"""CLI: structural (DOM) ACTION rung vs the visual ladder under render drift.
+
+Reproduces the desktop-benchmark shape that reframed the thesis from
+"vision-only" to "deterministic compiled automation with visual FALLBACK"
+(structural 21/21 vs compiled visual replay 6/21). The reusable, tested logic
+lives in :mod:`openadapt_flow.validation.structural_action`; this script is a
+thin runner that prints the ratios and writes ``structural_action.json``.
+
+Usage:  python benchmark/structural_action/structural_action_probe.py [n] [--show]
+"""
+from __future__ import annotations
+
+import json
+import sys
+from pathlib import Path
+
+from openadapt_flow.validation.structural_action import run_probe
+
+
+def main(argv: list[str]) -> int:
+    n = next((int(a) for a in argv if a.isdigit()), 21)
+    report = run_probe(n=n, headless="--show" not in argv)
+    out = Path(__file__).with_name("structural_action.json")
+    out.write_text(json.dumps(report, indent=2))
+    print(f"structural (DOM) rung : {report['structural_ratio']} acted correctly")
+    print(f"visual ladder only    : {report['visual_ratio']} acted correctly")
+    drifted = [t for t in report["targets"] if t["drifted"]]
+    d_struct = sum(t["structural_ok"] for t in drifted)
+    d_vis = sum(t["visual_ok"] for t in drifted)
+    print(
+        f"under drift ({len(drifted)} targets): structural {d_struct}/{len(drifted)}"
+        f" vs visual {d_vis}/{len(drifted)}"
+    )
+    print(f"wrote {out}")
+    return 0
+
+
+if __name__ == "__main__":
+    raise SystemExit(main(sys.argv[1:]))

openadapt_flow/__main__.py

@@ -179,6 +179,12 @@ def _cmd_replay(args: argparse.Namespace) -> int:
                     state_verifier=(
                         appliance.state_verifier if appliance else None
                     ),
+                    # Normal replay prefers the deterministic structural rung.
+                    # ``--drift`` exists to DEMONSTRATE the visual healing ladder
+                    # on the bundled MockMed app, so it forces the visual floor
+                    # (structure would resolve the injected drift and there would
+                    # be nothing to heal -- the very thing the flag shows).
+                    use_structural=not bool(args.drift),
                 ).run(
                     workflow,
                     params=params,