From d1e9dfbfdb83663123ffc4639b8d8156f79c17bf Mon Sep 17 00:00:00 2001 From: Richard Abrich Date: Wed, 8 Jul 2026 17:56:34 -0500 Subject: [PATCH 1/5] feat: lead /compare with OpenEMR head-to-head, keep MockMed as reproducible anchor Replace the single MockMed stat block in the computer-use-agents section with two panels: the OpenEMR public-demo head-to-head (both arms 100%, 1.8x median latency, $0 vs $0.55/run at list, 0 vs ~24 model calls/run) as the primary result with its caveats disclosed inline, and the MockMed benchmark beneath it as the CI-reproducible anchor. Numbers from openadapt-flow benchmark/openemr/results.json (2026-07-08). Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01CKrVJJy5jWVCkXAqgUqtqZ --- pages/compare.js | 69 +++++++++++++++++++++++++++++++++++++++--------- 1 file changed, 57 insertions(+), 12 deletions(-) diff --git a/pages/compare.js b/pages/compare.js index 05b2794..4645833 100644 --- a/pages/compare.js +++ b/pages/compare.js @@ -154,20 +154,29 @@ export default function ComparePage() {

- We measured it + We measured it on a real EMR

+

+ An 18-step add-patient-note workflow on the official + OpenEMR public demo, run both ways and judged by one + arm-independent OCR check, with a distinct + parameterized note per run. Both arms succeeded every + time: 20/20 compiled, 10/10 for a Claude computer-use + agent. The agent doesn't fail here — the + difference is what each run costs. +

- 7.7× faster + 1.8× faster

- median run: 4.9s compiled vs 37.5s agent + median run: 39.2s compiled vs 70.4s agent

- $0 vs $0.27 + $0 vs $0.55

model cost per run, at list price @@ -175,25 +184,61 @@ export default function ComparePage() {

- 5.1s vs 43.4s + 0 vs ~24

- 95th-percentile latency + model calls per run

- Same task, same success check, 100 compiled replays - against 20 runs of a Claude computer-use agent: both - arms passed 100% on our simple demo app, so the - difference isn't success rate, it's what - repetition costs. + Run the task 500 times and the ratios compound: about + $275 and ten hours of wall clock through the agent, + versus $0 and about five and a half hours compiled, + with every action auditable against the demonstrated + script. The compiled arm's price of entry is a + one-minute human demonstration; the agent needs only a + prompt. +

+

+ Caveats, disclosed up front: the OpenEMR demo is a + shared public instance that anyone can modify and that + resets daily, so this is a field result, not a + CI-reproducible one. The agent arm is N=10 (agent runs + cost real money and real load on a shared service), so + its 100% carries wide error bars. Results are pinned + to claude-sonnet-5 with the computer_20251124 tool on + 2026-07-08; newer models will differ. The OCR success + check errs conservative on dense EMR text and is + identical for both arms. +

+ + OpenEMR methodology and raw data + +
+ +
+

+ The reproducible anchor +

+

+ Because the OpenEMR numbers depend on a live shared + instance, we keep the same head-to-head on MockMed, + the demo clinic app that ships with openadapt-flow, + as the benchmark anyone can rerun deterministically: + 100 compiled replays against 20 agent runs, both arms + 100%, 4.9s vs 37.5s median, $0 vs $0.27 per run at + list price. Same orchestrator, same agent harness, + same style of OCR check.

- Methodology and raw data + MockMed methodology and raw data
From 79617292a1c31ec83e209a6810843c2fad4f5789 Mon Sep 17 00:00:00 2001 From: Richard Abrich Date: Wed, 8 Jul 2026 18:03:48 -0500 Subject: [PATCH 2/5] feat: disclose compiled self-flagged run in OpenEMR panel caveats One of the 20 compiled runs self-flagged postcondition drift on the final step and was verified saved by the arm-independent OCR check; add one sentence to the panel's caveat footnote so the page matches the canonical BENCHMARK.md disclosure. Headline numbers unchanged. Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01CKrVJJy5jWVCkXAqgUqtqZ --- pages/compare.js | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/pages/compare.js b/pages/compare.js index 4645833..a0440a5 100644 --- a/pages/compare.js +++ b/pages/compare.js @@ -206,7 +206,11 @@ export default function ComparePage() { resets daily, so this is a field result, not a CI-reproducible one. The agent arm is N=10 (agent runs cost real money and real load on a shared service), so - its 100% carries wide error bars. Results are pinned + its 100% carries wide error bars. One compiled run + self-flagged postcondition drift on the final step and + was verified saved by OCR; success is judged by the + arm-independent check for both arms, never + self-report. Results are pinned to claude-sonnet-5 with the computer_20251124 tool on 2026-07-08; newer models will differ. The OCR success check errs conservative on dense EMR text and is From 69d4797cb4fda2fefe08f0dd49bcbab88fa797b6 Mon Sep 17 00:00:00 2001 From: Richard Abrich Date: Fri, 10 Jul 2026 00:24:02 -0400 Subject: [PATCH 3/5] feat: add hybrid drift-fallback result to /compare MockMed anchor panel Compiled-first with agent fallback on detected halt: 20/20 at ~1/8 the agent's cost per successful run under injected drift, with caveats (synthetic detected-halt drift, assumed drift mix) noted inline and a link to the hybrid benchmark methodology. Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01CKrVJJy5jWVCkXAqgUqtqZ --- pages/compare.js | 29 +++++++++++++++++++++++------ 1 file changed, 23 insertions(+), 6 deletions(-) diff --git a/pages/compare.js b/pages/compare.js index a0440a5..7197669 100644 --- a/pages/compare.js +++ b/pages/compare.js @@ -238,12 +238,29 @@ export default function ComparePage() { list price. Same orchestrator, same agent harness, same style of OCR check.

- - MockMed methodology and raw data - +

+ On the same harness under injected UI drift, a hybrid + mode — compiled replay first, agent fallback + only on a detected halt — matched agent + reliability (20/20) at roughly one-eighth the + agent's cost per successful run. Details and + caveats (synthetic detected-halt drift, assumed drift + mix) in the repo. +

+

From 0712942e36238676324ce3d3430634f6d90183f9 Mon Sep 17 00:00:00 2001 From: Richard Abrich Date: Fri, 10 Jul 2026 14:29:51 -0400 Subject: [PATCH 4/5] feat: reframe /compare spine to measured silent wrong-action rate Lead the page with the instrument thesis: self-healing GUI replay can silently write to the wrong record under UI drift, nobody measures it, we red-teamed our own engine five times (frozen corpus, FA 0.000% / ~26-28% safe-halt cost) and found two shipping competitors each 3/3 silent wrong-patient under identity drift. Add the honest DOM-selector exception. Benchmarks (OpenEMR/MockMed/ hybrid) reframed as the supporting efficiency case, not the lead. Prominent honest-limitations link to LIMITS.md plus the adversary log and competitor study. Build green. Co-Authored-By: Claude Opus 4.8 Claude-Session: https://claude.ai/code/session_01CKrVJJy5jWVCkXAqgUqtqZ --- pages/compare.js | 144 +++++++++++++++++++++++++++++++++++++++++++---- 1 file changed, 134 insertions(+), 10 deletions(-) diff --git a/pages/compare.js b/pages/compare.js index 7197669..1e26428 100644 --- a/pages/compare.js +++ b/pages/compare.js @@ -9,7 +9,7 @@ const webPageSchema = { name: 'How OpenAdapt compares', url: 'https://openadapt.ai/compare', description: - 'An honest comparison of OpenAdapt with traditional RPA platforms, AI computer-use agents, and browser recording tools: how automations are built, what each run costs, and what happens when the UI changes.', + 'Every self-healing GUI automation tool can silently write to the wrong record under UI drift. We built the harness that measures it, red-teamed our own engine five times until it stopped, and pointed it at the category. How OpenAdapt compares to RPA, AI computer-use agents, and browser recorders on safety first, then cost and coverage.', isPartOf: { '@type': 'WebSite', name: 'OpenAdapt.AI', @@ -75,13 +75,13 @@ export default function ComparePage() { How OpenAdapt compares to RPA, AI agents, and browser recorders | OpenAdapt