Skip to content

apex_sim: production-realistic bench conditions + score-discrimination (AUC)#69

Closed
jspaezp wants to merge 5 commits into
apex-weighting-cleanfrom
apex-sim-realism
Closed

apex_sim: production-realistic bench conditions + score-discrimination (AUC)#69
jspaezp wants to merge 5 commits into
apex-weighting-cleanfrom
apex-sim-realism

Conversation

@jspaezp

@jspaezp jspaezp commented Jul 19, 2026

Copy link
Copy Markdown
Collaborator

Stacked on #68 (apex-weighting-clean). Bench-only (apex_sim); no timsseek
production changes.

Makes apex_sim benchmark the apex-finder and the score as SEPARATE functions
under production-realistic conditions, and adds a signal-vs-noise score metric.
All additive: existing canonical suite / SimParams behavior preserved by
defaults.

Why

The old bench conflated two jobs and tested neither the way production runs them:

  • Pass 1 (apex finder, weighted apex_profile) runs over a BROAD ~1695-cyc window
    (Phase 1) and inside a NARROW ~150-cyc calibrated window (Phase 3).
  • Pass 2 (score / main_score) is meant to separate real signal from noise, but
    the bench only measured apex-cycle recovery, never score magnitude.
    It also pinned the apex to an integer index at the SAME position every run, and
    used a fixed interferent COUNT (so a wider window = lower density = easier).

Empirical anchors (derived from a real run)

250225_Desnaux_200ng_Hela ICC_off, default tol. Instrumented the two callsites

  • read a persisted results.parquet (11743 targets at q<0.01):
  • broad window (Phase 1): 1695 cycles (RT-unrestricted, whole gradient).
  • narrow window (Phase 3): 82-206 cycles, median 152.
  • peak width: rising/falling ~3 cyc => sigma ~1 (sim's 1.0 already realistic).

Changes

  1. Density-scaled interferents: RandomPeaks.density_per_cycle (+ hardness)
    => count = round(density * n_cycles * hardness). Fixed density as the window
    grows; hardness>1 over-loads on purpose. count path preserved.
  2. Sub-cycle apex jitter: SimParams.apex_jitter=Some(range) draws the realized
    apex = apex_cycle + U(-range,range) + U(-0.5,0.5) per seed (off-grid, moving
    target); SimData.realized_apex_cycle reports it and run_sensitivity scores
    against it. None keeps the historical on-grid fixed apex byte-identical.
  3. ROC-AUC discrimination: run_discrimination scores matched present/absent
    pairs (absent = all fragments AND precursor zeroed, same seed => identical
    noise+interferents) and reports ROC-AUC of the Pass-2 main_score.
  4. Realistic suites: broad_suite (1695 cyc) + narrow_suite (150 cyc), density
    0.41/cyc (= canonical 100/245, so window size is the only changed variable),
    jittered apex, per-scenario stressors incl a 3x-density hard variant.
  5. Example prints canonical / broad / narrow recovery tables + a narrow AUC
    table. Canonical header kept as === summary so bench_out/score.sh still
    parses it. Each scenario is its own line (comment out to disable).

Results (n=2000, broad n=500, tol +/-2)

Same density, wider window is measurably harder:

high_noise+interf canonical 245cyc 86% broad 1695cyc 54% narrow 150cyc 90%
mismatched canonical 88% broad 59% narrow 93%

hard_3x_density (1.23/cyc) collapses recovery to ~0-8% (explicit stressor).

Score discrimination (narrow AUC, present vs pure noise):

clean 1.000 moderate 0.870 high_noise 0.840 mismatched 0.876 hard 0.680

med(present) > med(absent) in every scenario. The score separates well until
interference piles up, then degrades to ~0.68 -- the axis the old bench could
not see.

Follow-up (out of scope, bench-first)

Pass-2 apex relocation: score_at computes effective_apex from find_joint_apex on
the UN-weighted cosine_profile (global argmax), which drifts onto interferents
in narrow windows. The narrow-recovery + AUC metrics here exist to prove that
fix when it lands.

Validation

  • cargo test -p apex_sim: 11 passed (density scaling, jitter determinism/sub-
    cycle, roc_auc, discrimination).
  • cargo build -p apex_sim --release: clean.
  • Full bench: bench_out/logs/realistic-bench-v2.log.
  • Note: apex_sim has a pre-existing clippy field_reassign_with_default idiom in
    the suite base() closures; the new suites follow it for consistency.

jspaezp added 5 commits July 18, 2026 18:54
Fixed count in a wide window understates interferent density and makes
apex-finding artificially easy. density_per_cycle > 0 overrides count as
round(density * n_cycles * hardness); defaults keep count-based behavior.
apex_jitter=Some(range) draws realized apex = apex_cycle + U(-range,range)
+ U(-0.5,0.5) per seed (off-grid, moving target). None keeps the historical
on-grid fixed apex byte-identical. run_sensitivity scores each run against
its realized apex.
roc_auc (average-rank Mann-Whitney) + run_discrimination: matched pairs
share a seed; the absent twin zeros BOTH fragments and precursor (true
no-signal), so identical noise+interferents isolate the real peak's effect
on the Pass-2 score. Measures the score as a signal/noise separator.
…bles

Density-scaled interferents + jittered sub-cycle apex at production window
sizes. Example prints canonical/broad/narrow recovery tables plus a narrow
score-discrimination AUC table. Canonical header kept as '=== summary' so
score.sh still parses it.
…0/245)

2.5/cyc (mis-derived from the 40-cyc viewer example) drowned every noisy
scenario at realistic window sizes. 0.41/cyc matches the canonical suite's
interferent density, isolating window size as the changed variable: broad
high-noise recovery 86%(245cyc) -> 54%(1695cyc); narrow AUC 1.0 clean ->
0.68 heavy. hardness=3 remains as an explicit stressor.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant