feat: DOM-selector benchmark arm — the incumbent comparison

[abrichr]

Why

External criticism of this repo's benchmarks, which we accept: the real incumbent for "run the same GUI workflow 500 times" is not a computer-use agent — it is a DOM-selector script (Playwright/Selenium). Also $0, also fast, no OCR anywhere. Instead of arguing, this PR runs that experiment head-to-head against the compiled vision replay — same task, same frozen drift schedule, same arm-independent OCR + identity success check (verify_hybrid_final, reused from the hybrid benchmark), wrong-action events measured for every arm.

The task spec ("open the FIRST referral task") underdetermines exactly one selector, so both readings run as steelman arms: DOM (positional) = first row's Open button (literal reading); DOM (name-filtered) = the Open button in the row named Jane Sample (identity reading, hardcoding the demonstrated patient exactly as the compiled arm's recorded identity band does). Full generated report: benchmark/dom/BENCHMARK.md.

Three-arm perturbation matrix (n=2 each, deterministic, $0)

drift mode	compiled replay	DOM (positional)	DOM (name-filtered)
lookalike	halt (fail-closed)	WRONG ACTION (#patient/p0)	success
missing	halt (fail-closed)	WRONG ACTION (#patient/p2)	halt (fail-closed)
grow	halt (fail-closed)	WRONG ACTION (#patient/g1)	success
sort	halt (fail-closed)	WRONG ACTION (#patient/p2)	success
theme	mixed*	success	mixed*
rename	success (2 heals)	halt — needs human edit	halt — needs human edit
move	success (2 heals)	success	success
typelabel	success (1 heal)	success	success

* one theme run per affected arm completed correctly (#patient/p1, right patient) but failed the dark-palette OCR judge — a judge false negative, disclosed, counted as a failure, and the two disputed final screenshots are committed alongside the report.

Frozen 20-slot schedule: all three arms tie 14/20 (14 clean, 0/6 drifted — notice/reqfield/modal-once stop every arm). Wrong-action totals: compiled 0, DOM-positional 8, DOM-name-filtered 0. Maintenance totals: compiled 0, DOM-positional 8, DOM-name-filtered 10. Both DOM arms are ~36x faster per clean run (p50 0.2s vs 7.2s).

Verdict (verbatim from the generated report)

The wrong-action vector is spec underspecification, not "Playwright". On the frozen schedule all arms tie (compiled 14/20; dom 14/20; dom_named 14/20): the drift that halts the compiled replay (notice/reqfield/modal-once) stops both DOM scripts too. Both DOM scripts are ~36x faster per clean run (p50 0.2s vs 7.2s). The perturbation matrix is where the arms separate, and they separate by HOW EACH ARM NAMES ITS TARGET, not by DOM vs vision. (a) Positional selectors silently retarget under data drift. The positional script ("first row" — the literal reading of the task spec) wrote to the WRONG PATIENT on 4 of 8 modes (grow, lookalike, missing, sort; 8 runs, every one with a healthy-looking final screen). A position-phrased spec cannot notice that the row's identity changed. (b) The identity reading fixes it — and where a stable DOM exists, it also out-completes the compiled arm. The name-filtered script (same code, one selector keyed to the demonstrated patient) completed CORRECTLY on grow, lookalike, move, sort, typelabel and failed closed on missing, rename, with zero wrong actions. The compiled arm was equally safe (wrong actions: 0) but never healed to the true row on these modes — every data-drift outcome was a halt: on data drift the name-filtered DOM arm finished the work the compiled arm safely declined. (c) What demonstration buys: the identity came for free. Nobody had to DECIDE that "first referral" really means Jane Sample — the demonstration captured the target's identity as a matter of course, while the DOM arms needed that judgment hand-written into a selector (and the positional variant shows what happens when it is not). The compiled arm's remaining browser-side edges on this data: demo-derived identity with no spec authoring, heal-through of label drift (rename broke both DOM scripts' Open selector — a human edit each — while the ladder healed through it), and fail-closed halts with an accurate report. Its costs are equally plain: ~36x slower per run, and an OCR judge with failure modes of its own. (One condition — theme — produced judge-disputed verdicts: an OCR-judge artifact affecting both arms equally, not an automation difference; see the measurement-validity section.) Boundary, stated plainly: this comparison exists ONLY on browser backends. On desktop, VDI/Citrix, or any pixels-without-DOM substrate there is no selector script to write — the criticism's own point — and wherever a stable, accessible DOM exists, an identity-keyed selector script is the honest incumbent to beat: as fast as the positional one and, on this matrix, as safe as the compiled arm.

What's in the PR

• openadapt_flow/benchmark/dom_arm.py — both steelman scripts (positional + name-filtered, differing in exactly one documented selector), harness, outcome classification (success / halt-or-error / WRONG-ACTION / maintenance / verification-dispute), aggregation, three-column markdown renderer, three-arm chart with dispute footnote, orchestrator, CLI. Reuses the hybrid benchmark's frozen SCHEDULE and verify_hybrid_final.
• openadapt_flow/mockmed/static/app.js — flag-gated sort drift mode.
• benchmark/dom/{BENCHMARK.md, results.json, outcome_matrix.png} + the two disputed theme finals.
• .gitignore — ignores benchmark/dom/{finals/,rows.jsonl} (disputed finals force-added).
• Tests: tests/test_dom_arm.py (24) incl. the one-selector-difference and identity-scoping cases + tests/test_mockmed.py sort-mode tests. Full suite green.

Prior draft's asymmetric variant dismissal is retracted in the report: the compiled arm hardcodes the same patient constant in its recorded identity band; the arms differ in how identity is captured (demonstration vs hand-authored selector) and in fail-closed semantics, not in whether identity is encoded. The "writes to the wrong patient" framing is now scoped to positional selectors specifically. $0 spent: zero model calls anywhere.

🤖 Generated with Claude Code

https://claude.ai/code/session_01CKrVJJy5jWVCkXAqgUqtqZ