feat: automated Parallels desktop benchmark pipeline (Phase 2)

[abrichr]

What

A fully automated, programmatic, $0 desktop-benchmark pipeline for openadapt-flow Phase 2, driven end-to-end by prlctl + Python against the local Parallels Windows 11 ARM VM on Apple Silicon. No manual/GUI steps, no cloud, no model calls (the compiled-replay and UIA arms make zero). Apple Silicon has no nested virtualization, so the WAA/QEMU stack can't run here — Parallels (native ARM virtualization) is the substrate.

Proves the record → compile → replay mechanism on a real desktop app over the vision-only Phase-1 WindowsBackend contract, benchmarked against the desktop incumbent (UIA selectors), judged by database ground truth.

Layers (each validated before the next)

Layer	Status
prlctl exec + screenshot loop	✅ (prlctl exec runs as SYSTEM/session-0; solved with CreateProcessAsUser into session 1)
WindowsBackend full loop over in-guest WAA HTTP shim	✅ real desktop PNG in, click/type lands
DB-readable ground truth + seeded app	✅ (WinForms substitute for OpenDental — see below)
Compiled replay on desktop (identity bands on desktop text)	✅
Drift matrix + UIA arm	✅

Components

• openadapt_flow/backends/parallels_vm.py — ParallelsVM: lifecycle / snapshot / revert / exec / capture / guest+host IP / ephemeral-port file push / session-1 shim launch.
• scripts/desktop/session1_launch.py — SYSTEM→session-1 launcher (WTSQueryUserToken + CreateProcessAsUser), the foundational blocker solved.
• scripts/desktop/waa_shim.py — in-guest WAA-contract HTTP shim (/screenshot, /execute_windows, /uia).
• scripts/desktop/patient_notes.ps1 + pn_db.py — WinForms list-select→edit→save app + SQLite ground truth.
• scripts/desktop/uia_arm.py — pywinauto UIA incumbent (identity + positional, steelmanned).
• openadapt_flow/benchmark/desktop_benchmark.py — orchestrator: 3 arms × 7 conditions, DB judge, benchmark/desktop/{results.json,BENCHMARK.md,outcomes.png}.

Results (n=3/cell, DB ground truth)

Arm	success	wrong-action	safe-halt	false-abort
compiled (vision)	29%	14%	12	9
uia_identity	100%	0%	0	0
uia_positional	71%	29%	0	0

Honest verdict, both ways:

1. The record→compile→replay mechanism works on a real desktop with identity bands verified on desktop-rendered text, on a pixel substrate with no DOM.
2. Vision replay is defeated by render-scale (DPI proxy) and theme drift (0% completion) — but it halts, never mis-writes (pre-committed "DPI is ugly" result).
3. The positional UIA incumbent silently mis-writes on any name collision (decoy + siblings).
4. Identity catches a discriminable decoy (safe-halt) but false-verifies a near-lexical sibling (Sorenson≈Sorensen, adjacent DOB) → 3 wrong-actions: the browser-tuned matcher's OCR-jitter tier treats a 1-char/1-digit difference as noise. This is the desktop analogue of the open browser wrong-action findings — the safety claim does not yet fully transfer to desktop.
5. UIA-tree quality 5/6; the identity-critical patient row has no AutomationId — the measured "vision is necessary" evidence.

Walls (documented honestly)

• OpenDental was the intended target; its trial (TrialDownload-25-3-48.exe, 149 MB) is an interactive bootstrapper gated by SmartScreen + a UAC secure-desktop prompt (unreachable by session-1 pyautogui) with no silent flags → not no-touch installable. Substituted a real WinForms app with the same shape + exact SQL ground truth (labelled in every output). App choice is secondary; the deliverable is the automated pipeline.
• Caveats (ARM+x64-emulation rendering ≠ native x86; render-scale-as-DPI proxy; window-resize deferred) in docs/desktop/LIMITS.md.

Notes

• Includes the Phase-1 WindowsBackend + capture adapter (previously only on feat/desktop-backend), rebased onto current main.
• User VM safety: snapshotted before any mutation (pre-openadapt-phase2); a harness-ready running snapshot gives warm-boot + per-run reset. VM left suspended.
• Tests mock the VM/HTTP; full suite green (468 passed).

🤖 Generated with Claude Code

https://claude.ai/code/session_01CKrVJJy5jWVCkXAqgUqtqZ