Skip to content

docs: competitor drift study — silent wrong-action benchmark#18

Draft
abrichr wants to merge 6 commits into
mainfrom
feat/competitor-drift-study
Draft

docs: competitor drift study — silent wrong-action benchmark#18
abrichr wants to merge 6 commits into
mainfrom
feat/competitor-drift-study

Conversation

@abrichr

@abrichr abrichr commented Jul 10, 2026

Copy link
Copy Markdown
Member

What

Comparative benchmark: do OTHER self-healing/deterministic-replay browser automation tools silently write wrong state under UI drift, the way our own pre-fix replayer did (PR #12 / VALIDATION.md)?

Tools tested against our own local MockMed app, identical task and drift modes, arm-independent ground truth (client-state beacon, never trusting tool self-reports):

  • workflow-use (browser-use org, pinned 18d4613) — 3 arms: stock det, stock no-AI semantic, steelman (semantic + its own LLM-generated agent step)
  • Skyvern (pinned fd9c1eb, local server, SQLite) — run_with=code cached-script mode with ai_fallback + enable_self_healing
  • playwright codegen 1.61 (stretch, $0) — unedited emitted script

Headline

arm silent wrong-patient writes under lookalike/missing/grow
our pre-fix replayer (committed reference) 3/3
our post-fix replayer (committed reference) 0/3 (safe-halts)
workflow-use steelman 3/3 — claimed success; its own extraction printed the wrong patient's name
Skyvern cached script 3/3 — claimed completed; AI fallback never consulted (selector matched the imposter), LLM completion verification approved the wrong patient
workflow-use stock modes 0/3, but 0% availability (crash before the drift step even on baseline)
playwright codegen 0/3 — absorbed via MockMed's identity-bearing DOM ids (architecture artifact, documented)

Cosmetic drift (theme/rename/move): both LLM tools healed and passed; codegen safe-halted on rename.

Deliverables

  • docs/validation/COMPETITOR_STUDY.md — full matrix, per-tool mechanisms, versions/configs verbatim, spend accounting, fairness caveats (incl. that drift modes were designed against OUR replayer, and the "first row" intent-binding caveat)
  • scripts/competitor_study/ — reproducible harness: instrumented MockMed study server (URL-invisible drift injection + state beacon), verdict module, per-tool record/build/replay runners
  • Raw per-run evidence local + gitignored under runs/competitor_study/evidence/

Spend

$0.92 total at list price (cap $10, abort $8 — never approached). Exact API-reported token usage; conservative accounting for prompt-cache tokens.

Not merged intent

DRAFT for review — do not merge without review of the fairness framing.

🤖 Generated with Claude Code

https://claude.ai/code/session_01CKrVJJy5jWVCkXAqgUqtqZ

abrichr and others added 6 commits July 10, 2026 12:25
@abrichr @claude
feat: competitor drift-study harness and workflow-use runners
b54a610 
Instrumented MockMed study server (URL-invisible drift injection +
client-state beacon), arm-independent verdict module, and the
record/build/replay runners for workflow-use (pinned 18d4613).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CKrVJJy5jWVCkXAqgUqtqZ
@abrichr @claude
feat: skyvern and playwright-codegen runners for drift study
39f8240 
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CKrVJJy5jWVCkXAqgUqtqZ
@abrichr @claude
docs: competitor drift study — silent wrong-action results matrix
746aed1 
workflow-use (steelman arm) and Skyvern (cached-script mode) both
silently wrote wrong-patient encounters in 3/3 row-identity drift
modes while reporting success; playwright codegen absorbed row drift
via identity-bearing DOM ids. Full matrix, mechanisms, spend (/bin/zsh.92),
and fairness caveats.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CKrVJJy5jWVCkXAqgUqtqZ
@abrichr @claude
docs: revise competitor study per publishability review
a1e67b5 
- Regenerate spend table directly from spend_ledger.jsonl (exact per-group
  runs/calls/tokens; total $0.9367); footnote the anomalous cheap W3
  baseline (2 calls, 309/477) instead of averaging.
- Byte-exact all competitor quotes; commit the workflow-use lookalike agent
  transcript (Memory line naming Taylor Duplicate) and the generated Skyvern
  script to evidence; note extraction-label case differs between the two
  harness variants (both wrote to p0).
- Scope the 'missing is immune' claim to the recording-based arm; reframe
  Skyvern to the unrebuttable version (cache minted against p1; cache-mode
  replay binds nothing to that entity) and disclose the goal-text limitation
  (identity-naming goal unmeasured).
- Add macOS-reference-platform + platform-dependent-grow caveats wherever our
  own reference rows are cited.
- Fix beacon dedup key to exclude Date.now() (POST only on real state change,
  no longer defeats networkidle); verified verdict-neutral via $0 codegen
  matrix (7/7) + one Skyvern lookalike ($0.012, still silent-WA p0). Disclose
  the prior periodic-beacon behavior as non-fatal; mark wall times
  non-comparable.
- Reword ChatAnthropic provenance (README programmatic examples use
  ChatOpenAI via the same BaseChatModel interface).
- Commit sanitized evidence_summary.json (per-run digest + ledger, no keys).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CKrVJJy5jWVCkXAqgUqtqZ
@abrichr @claude
docs: byte-exact goal quotes; distinguish USER_GOAL (workflow-use) fr…
c547419 
…om YAML navigation_goal (Skyvern)

- Correct the claim that both compilers got study_common.USER_GOAL verbatim:
  workflow-use's builder did; Skyvern's navigation block got the committed
  YAML navigation_goal (different wording, same task, with an explicit
  completion criterion). Quote each in its native byte-exact form.
- Quote the Skyvern completion goal byte-exact from the YAML (capital O,
  full completion sentence) in the mechanism note.
- Mark the workflow-use task quote as a verbatim substring with elision.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CKrVJJy5jWVCkXAqgUqtqZ
@abrichr @claude
docs: fix Skyvern lookalike repeat count 2x->3x for internal consistency
de4f541 
The results matrix and mechanism note say lookalike reproduced 3/3
(baseline-drift + -rep2 + -beaconcheck, all in the ledger); the
runs-per-cell line said 2x. Align to 3x.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CKrVJJy5jWVCkXAqgUqtqZ
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@abrichr