feat: competitor-drift instrument harness (pluggable external-agent silent-wrong-action-rate runner, cost-capped)

[abrichr]

What

Extends the self-directed silent-wrong-action benchmark (#67) from our own runtime to any external computer-use agent. A new openadapt_flow.instrument package points the #63 EffectVerifier at an arbitrary external agent's runs against the MockMed transactional-fault suite (mockmed.fault_server) and measures its silent-wrong-action rate — a wrong / absent / duplicate business effect that landed while the agent reported success — output anonymized by architecture class (Tool A/B/C), never a vendor name.

This PR is the HARNESS ONLY. It spends $0, makes no model / paid API call, ships no concrete competitor adapter, and names no vendor. The real, cost-capped run against a real competitor is a separate, user-gated decision — this makes it one command away.

The seam: ExternalAgentAdapter Protocol

class ExternalAgentAdapter(Protocol):
    architecture_class: str                     # anonymized "Tool A/B/C" — never a vendor
    def estimate_cost_usd(self, task) -> float: ...           # pre-flight projection
    def run_task(self, task, target_url, *, max_steps) -> AgentRunResult: ...

A real adapter (a thin wrapper over a vendor's own record/replay/agent entry points) plugs in later, out of this repo — the Protocol is the only contact point, and the vendor's code never touches our scoring path. A worked wrapping example is in the ExternalAgentAdapter docstring. The harness reads the MockMed system of record itself with RestRecordVerifier; the only agent-supplied signal the metric uses is the agent's own success claim.

Cost guardrails (mandatory)

CostGuard is a hard kill-switch on max_cost_usd / max_steps / max_runs that aborts the whole run the instant a cap would be crossed:

• pre-flight (can_start): refuses to start a run whose estimate would cross the cost cap, or once max_runs is hit;
• post-flight (record): books actual spend/steps and aborts everything if cumulative cost passed the cap or a run blew the step budget;
• dry-run mode: projects total cost from the adapter's estimates and returns without running or spending;
• every abort logs what was spent and how many runs were dropped. No run can silently exceed the cap.

Stub-adapter proof (CI, $0, offline)

StubExternalAgentAdapter emulates an external agent driving MockMed:

• screen_blind (a vision-style blind agent) → silent-wrong-action rate 100% on each of the 5 silent fault classes (partial / optimistic / duplicate / double / stale), 0% on the clean classes (ok / idempotent), and correctly 0% on session (safe halt) and timeout (write actually landed). Headline: 55.6% (5/9).
• honest (negative control) → 0% everywhere, proving the metric is not hardwired to fire.

For every silent-wrong row, our EffectVerifier refused to confirm — i.e. pairing the agent with our verifier converts the silent wrong-action into a safe halt.

Tests

25 tests, all green (tests/test_competitor_drift_instrument.py): nonzero-on-faults / zero-on-clean, the honest negative control, cost/step/run-cap aborts, dry-run projection, and anonymization (vendor labels rejected structurally; rendered artifacts scanned for vendor strings). Reuses the #67 ground-truth judge and #63 effect contract as the single source of truth. No existing files modified — additive only.

Run it: python -m openadapt_flow.instrument.competitor_drift --dry-run (estimate) or --stub-mode screen_blind ($0 proof).

🤖 Generated with Claude Code

https://claude.ai/code/session_01CKrVJJy5jWVCkXAqgUqtqZ