Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -15,3 +15,5 @@ benchmark/hybrid/finals/
benchmark/hybrid/rows.jsonl
benchmark/dom/finals/
benchmark/dom/rows.jsonl
benchmark/desktop/finals/
benchmark/desktop/_work/
42 changes: 42 additions & 0 deletions benchmark/desktop/BENCHMARK.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,42 @@
# Desktop Benchmark (Phase 2) — compiled vision replay vs UIA incumbent

_Generated 2026-07-10T21:48:06.669583+00:00_

**Task.** Patient Notes (WinForms) search -> select -> note -> save; DB-ground-truth judge; $0 (no model calls)

**Substrate.** Parallels Windows 11 ARM VM on Apple M2 Max; WindowsBackend over in-guest WAA HTTP shim (session 1)

> WinForms substitute for OpenDental (trial not no-touch installable; see PHASE2.md).

## Headline

| Arm | n | success | wrong-action | safe-halt | false-abort | success rate | wrong-action rate |
|---|--:|--:|--:|--:|--:|--:|--:|
| `compiled` | 21 | 6 | 3 | 12 | 9 | 29% | 14% |
| `uia_identity` | 21 | 21 | 0 | 0 | 0 | 100% | 0% |
| `uia_positional` | 21 | 15 | 6 | 0 | 0 | 71% | 29% |

## Identity transfer to desktop-rendered text

- Compiled-arm **armed coverage**: 2/4 click steps carry an identity band (50%).
- UIA-tree quality: 5/6 workflow targets expose a usable AutomationId (83%); the identity-critical patient row does **not** (`identity_target_has_id=False`) — the measured 'vision is necessary' evidence.

## Outcome matrix (per arm × condition)

| Arm | clean | render_125 | render_150 | theme_dark | data_reorder | data_decoy | data_siblings |
|---|---|---|---|---|---|---|---|
| `compiled` | 3/3✓ | 0/3✓ 3⚠abort | 0/3✓ 3⚠abort | 0/3✓ 3⚠abort | 3/3✓ | 0/3✓ | 0/3✓ 3✗wrong |
| `uia_identity` | 3/3✓ | 3/3✓ | 3/3✓ | 3/3✓ | 3/3✓ | 3/3✓ | 3/3✓ |
| `uia_positional` | 3/3✓ | 3/3✓ | 3/3✓ | 3/3✓ | 3/3✓ | 0/3✓ 3✗wrong | 0/3✓ 3✗wrong |

## Reading

- **Success** = the right patient got the right note and no one else did (DB ground truth). **Wrong-action** = a note landed on a different patient (silent mis-write). **Safe-halt** = the arm stopped without writing. **False-abort** = a safe-halt on a purely cosmetic condition (render-scale/theme) where the target was still present.
- Caveats (ARM+x64 emulation rendering, render-scale-as-DPI proxy, WinForms substitute for OpenDental) are in `docs/desktop/LIMITS.md`.

## Verdict (honest, both ways)

1. **The mechanism exists on desktop.** Record → compile → replay of a real WinForms workflow runs deterministically over the vision-only `WindowsBackend`, judged by DB ground truth — on a pixel substrate with no browser DOM. Identity bands are extracted and verified on **desktop-rendered** text.
2. **Vision replay is defeated by render-scale and theme drift** (render_125/150 and theme_dark → 0% success, all safe-halts / false-aborts). This is the pre-committed 'DPI is ugly' result and the roadmap justification for multi-scale / appearance-invariant matching. It **never mis-wrote** under cosmetic drift — it halted.
3. **The positional UIA incumbent silently mis-writes** under *any* name-collision drift (decoy and siblings) — the exact wrong-action the identity work targets, measured on the incumbent.
4. **Identity verification catches a *discriminable* decoy** (distinct surname/DOB → compiled safe-halts) **but false-verifies a *near-lexical* sibling** (Sorenson≈Sorensen, adjacent DOB → compiled wrong-action): the browser-tuned matcher's OCR-jitter similarity tier (`TOKEN_SIM_RATIO`) treats a 1-char surname / 1-digit DOB difference as noise. This is the **desktop analogue of the open browser wrong-action findings** — the safety claim does **not** yet fully transfer to desktop; a stricter/desktop-tuned matcher (or an exact-field key for high-collision fields) is required. UIA-identity distinguishes the same sibling only because it does exact cell-text equality — an option that vanishes on a broken-a11y or pixel-only substrate, where the vision matcher is the only lever.
Binary file added benchmark/desktop/outcomes.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Loading