feat: structural (DOM/UIA) action rung — vision-first, not vision-only

[abrichr]

Summary

Makes structural (DOM/accessibility) evidence a first-class ACTION rung — the deterministic top of the resolution ladder — instead of only an identity signal. On structure-bearing backends the runtime now re-finds the recorded target as a DOM/UIA element and acts on its center deterministically; it falls back to the visual ladder (template → ocr → geometry → grounder) only where structure is absent (pixel-only substrates: RDP/Citrix/canvas).

This is the thesis refinement from "vision-only" to "deterministic compiled automation with visual FALLBACK". Two external reviews + the literature converge here; the desktop benchmark showed UIA execution 21/21 vs compiled visual replay 6/21 under render drift.

Capability ladder (as implemented)

API → tool/MCP → **DOM/UIA (structural)** → template → template_global → ocr → geometry → grounder(VLM) → human

(API/tool-MCP are future placeholders.) structural is the new rung 0, tried FIRST when the backend implements StructuralActionBackend and the anchor carries a StructuralLocator. It sits ABOVE ocr, so the irreversible-risk gate lets an irreversible step act on it (strongest evidence). Rungs 1–5 are unchanged — the visual FALLBACK floor for pixel-only substrates (the healthcare/Citrix wedge).

What changed

• ir.py — new StructuralLocator (selector / role+name / UIA AutomationId) captured on Anchor.structural; new StructuralHandle; "structural" added to Rung.
• backend.py — new optional StructuralActionBackend protocol: structural_locator_at (record) + locate_structural (replay).
• resolver.py — structural rung runs FIRST; on miss/exception/pixel-only it falls through to the visual ladder unchanged. is_below_ocr("structural") == False.
• playwright_backend.py — DOM locate via stable #id / role+name, with an occlusion hit-test (Playwright actionability): a covered element (modal / click-shield) returns None so the occlusion-safe-halt invariant is preserved.
• windows_backend.py — UIA ControlFromPoint locator capture + AutomationId/role+name locate via the WAA execute channel.
• recorder.py / compiler/compile.py — capture the locator at record time and store it on the anchor (visual anchor kept as fallback).
• replayer.py — structural resolution flows through the SAME click path, so the identity gate + risk gate still fire on it; structural is exempt from healing (deterministic locate ≠ stale template). New use_structural flag (default True) lets the visual-floor characterization suites exercise the pixel-only path deliberately.

Availability improvement (measured)

benchmark/structural_action/ reproduces the framing on a real DOM with the real ladder: structural 21/21 vs visual 6/21 (under drift: 15/15 vs 0/15). Numbers are measured, never hardcoded.

Safety — identity/risk gating intact

• A structurally-resolved point still faces verify_target_identity; a sibling/glyph-collapse mismatch still aborts (tests/test_structural_rung.py::test_structural_resolution_still_faces_identity_gate). Structure makes identity stronger (exact element), never bypasses it.
• Occlusion-safe-halt preserved (covered target → structural returns None → visual floor also fails → halt, no click).
• Headline no-wrong-write property holds with structural engaged across a cosmetic-drift sweep (tests/e2e/test_structural_action.py).

Tests

• tests/test_structural_rung.py — resolver mechanics, risk gate, identity gate on structural, Windows UIA (faked WAA), Playwright DOM round-trip, availability probe.
• tests/e2e/test_structural_action.py — default happy path resolves all steps via structural (0 heals, 0 model calls); floor-vs-structural rung contrast; no-wrong-write under drift.
• Visual-floor characterization suites unchanged (run with use_structural=False).

🤖 Generated with Claude Code

https://claude.ai/code/session_01CKrVJJy5jWVCkXAqgUqtqZ