fix: identity matcher rebuilt against a frozen adversarial corpus (3rd wrong-patient reopening) + armed-coverage metric

[abrichr]

fix: identity matcher rebuilt against a frozen adversarial corpus — near-name siblings (3rd wrong-patient reopening) + armed-coverage metric

The vulnerability (confirmed, reproduced)

band_match returned (coverage=1.0, residue=0) — VERIFIED — for all of:

• Belford, Phil 1985-03-12 M vs Belford, Philip 1985-03-12 M (containment tier: 'Phil' ⊂ 'Philip')
• the reverse direction (similarity tier, ratio 0.8)
• Smith, John ... vs Smith, Joan ... (similarity tier: 0.75 ≥ 0.7)
• Belford, Phil ... vs Belford, Phillipa ... (containment tier)

This is the wrong-patient P0 reopening a third time (pixel-lookalikes → bands; residue-blind coverage + param disarm → token matcher + residue cap; now near-name siblings). Real EMR rows are full of near-name siblings, and downstream note verification does NOT catch a wrong-patient write — the note really is saved, in the wrong chart.

1. Held-out corpus FIRST, frozen before any evaluation or fix

To break the fix-against-the-probe cycle, commit 1 of this branch adds openadapt_flow/validation/adversary_corpus.py: a deterministic seeded generator (seed 20260710) of 4360 labeled pairs — 2200 different_entity (prefix-extension names, single-letter edits, transpositions, Jr/Sr/II suffixes, same-surname/-first siblings, shared clinical text, DOB off-by-one-field, MRN digit swaps, adjacent-row mixtures) and 2160 same_entity (OCR confusions, token splits/joins, dropped short tokens, case/whitespace jitter, segment reordering, occlusion, spurious tokens, compound noise). The sha256 manifest is committed and pinned by tests, so post-hoc tuning of the corpus toward the matcher is detectable in git history. The generator was not modified after first evaluation (no generator bugs found; byte-identical to the pre-evaluation commit).

2. Measured, then fixed, operating point from the ROC

Current (pre-fix) matcher on the corpus: 53.9% false-accept overall. By category: DOB off-by-one 99.1%, Jr/Sr 99.1%, single-letter edits 98.2%, transpositions 95.5%, prefix extensions 72.3%, MRN swaps 50.0%, same-surname 15.5%. False-abort 12.1%.

The rebuild (runtime/identity.py): tokens match ONLY on OCR-equivalence (canonical identity under real engines' confusion classes: l/1/i, O/0, 5/s, 2/z, 8/b, 9/g, rn/m, cl/d, vv/w) plus full-consumption splits/joins — the containment and 0.7-similarity tiers are gone. Unmatched tokens split into absence (uncovered runs, budgeted as before) and contradiction (near-miss ≥ 0.62 on canonical forms, semantic containment with alphabetic residue, replacement by an unexplained observed token, generational suffix on one side) with a zero budget.

Post-fix: 0.000% false accepts (0/2200), 10.69% false aborts (better than before on both axes). ROC swept over contradiction_sim × coverage × run_cap × contradiction_cap; chart + tables + raw sweep committed under docs/validation/ (identity_roc.png, IDENTITY_ROC.md, identity_roc.json).

Operating point, weighting said out loud (false accept = wrong-patient write, catastrophic; false abort = ~$0.10 hybrid fallback; priced at 4+ orders of magnitude): coverage 0.8 / run_cap 4 / sim 0.62 / contradiction_cap 0. Deliberately NOT the Pareto-min-false-abort corner (0.7/8, FAbort 7.96%): that corner's zero rests entirely on the contradiction rule (evade it → FA 60.8%), while at 0.8/4 the older budgets independently stop 79.5% — defense in depth, bought with 2.7pp of false aborts concentrated in occlusion shapes whose identity tokens were never read (refusing those is correct).

Pins: all four sibling probes as permanent mismatch tests; operating-point boundary tests; a corpus-wide zero-false-accept regression test (a rate, not a probe list) + 12% false-abort budget. True-row live shapes (OpenEMR modal permutation, OCR jitter, splits/joins, MockMed fixtures) still verified.

3. Protection coverage as a first-class metric

Identity verification covers ONLY armed steps — live OpenEMR checks armed 4/12; a fresh MockMed demo bundle arms 1/8. Now auditable everywhere:

• workflow.json: per-step identity_armed / identity_unarmed_reason (concrete reasons: no readable band text / only the target's own label / too generic after volatile filtering) — auditable before running.
• REPORT.md: "N of M click steps identity-armed" + unarmed steps by id/intent/reason, computed over the whole bundle at run start. Verified end-to-end via CLI record→compile→replay.
• Benchmark generators (MockMed + OpenEMR + hybrid): compiled-arm rows/aggregates carry the metric; methodology sections render it. Committed BENCHMARK.md files' results.json predate the metric and carry an explicit "not captured in this results.json" note (no fabricated numbers).
• docs/LIMITS.md: the dangerous list now LEADS with the coverage gap; wrong-entity section updated for the rebuild.
• docs/validation/VALIDATION.md: 2026-07-10 fix update — third reopening said plainly with probe strings, corpus methodology, before/after per category, ROC + operating point rationale, coverage surfaces.

Verification

• Full suite green post-merge with main (unit + perturbation/chaos/primitives/record-compile-replay e2e): see checks.
• Zero Anthropic API calls anywhere (ANTHROPIC_API_KEY unset throughout).
• Real-flow verification: CLI demo-record → compile → replay produces REPORT.md with the coverage section and per-step reasons; bundle JSON carries the audit fields.

🤖 Generated with Claude Code

https://claude.ai/code/session_01CKrVJJy5jWVCkXAqgUqtqZ