apex_sim: production-realistic bench conditions + score-discrimination (AUC)#69
Closed
jspaezp wants to merge 5 commits into
Closed
apex_sim: production-realistic bench conditions + score-discrimination (AUC)#69jspaezp wants to merge 5 commits into
jspaezp wants to merge 5 commits into
Conversation
Fixed count in a wide window understates interferent density and makes apex-finding artificially easy. density_per_cycle > 0 overrides count as round(density * n_cycles * hardness); defaults keep count-based behavior.
apex_jitter=Some(range) draws realized apex = apex_cycle + U(-range,range) + U(-0.5,0.5) per seed (off-grid, moving target). None keeps the historical on-grid fixed apex byte-identical. run_sensitivity scores each run against its realized apex.
roc_auc (average-rank Mann-Whitney) + run_discrimination: matched pairs share a seed; the absent twin zeros BOTH fragments and precursor (true no-signal), so identical noise+interferents isolate the real peak's effect on the Pass-2 score. Measures the score as a signal/noise separator.
…bles Density-scaled interferents + jittered sub-cycle apex at production window sizes. Example prints canonical/broad/narrow recovery tables plus a narrow score-discrimination AUC table. Canonical header kept as '=== summary' so score.sh still parses it.
…0/245) 2.5/cyc (mis-derived from the 40-cyc viewer example) drowned every noisy scenario at realistic window sizes. 0.41/cyc matches the canonical suite's interferent density, isolating window size as the changed variable: broad high-noise recovery 86%(245cyc) -> 54%(1695cyc); narrow AUC 1.0 clean -> 0.68 heavy. hardness=3 remains as an explicit stressor.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #68 (apex-weighting-clean). Bench-only (apex_sim); no timsseek
production changes.
Makes apex_sim benchmark the apex-finder and the score as SEPARATE functions
under production-realistic conditions, and adds a signal-vs-noise score metric.
All additive: existing canonical suite / SimParams behavior preserved by
defaults.
Why
The old bench conflated two jobs and tested neither the way production runs them:
(Phase 1) and inside a NARROW ~150-cyc calibrated window (Phase 3).
the bench only measured apex-cycle recovery, never score magnitude.
It also pinned the apex to an integer index at the SAME position every run, and
used a fixed interferent COUNT (so a wider window = lower density = easier).
Empirical anchors (derived from a real run)
250225_Desnaux_200ng_Hela ICC_off, default tol. Instrumented the two callsites
Changes
RandomPeaks.density_per_cycle(+hardness)=> count = round(density * n_cycles * hardness). Fixed density as the window
grows;
hardness>1over-loads on purpose.countpath preserved.SimParams.apex_jitter=Some(range)draws the realizedapex = apex_cycle + U(-range,range) + U(-0.5,0.5) per seed (off-grid, moving
target);
SimData.realized_apex_cyclereports it and run_sensitivity scoresagainst it.
Nonekeeps the historical on-grid fixed apex byte-identical.run_discriminationscores matched present/absentpairs (absent = all fragments AND precursor zeroed, same seed => identical
noise+interferents) and reports ROC-AUC of the Pass-2 main_score.
0.41/cyc (= canonical 100/245, so window size is the only changed variable),
jittered apex, per-scenario stressors incl a 3x-density hard variant.
table. Canonical header kept as
=== summaryso bench_out/score.sh stillparses it. Each scenario is its own line (comment out to disable).
Results (n=2000, broad n=500, tol +/-2)
Same density, wider window is measurably harder:
high_noise+interf canonical 245cyc 86% broad 1695cyc 54% narrow 150cyc 90%
mismatched canonical 88% broad 59% narrow 93%
hard_3x_density (1.23/cyc) collapses recovery to ~0-8% (explicit stressor).
Score discrimination (narrow AUC, present vs pure noise):
clean 1.000 moderate 0.870 high_noise 0.840 mismatched 0.876 hard 0.680
med(present) > med(absent) in every scenario. The score separates well until
interference piles up, then degrades to ~0.68 -- the axis the old bench could
not see.
Follow-up (out of scope, bench-first)
Pass-2 apex relocation: score_at computes effective_apex from find_joint_apex on
the UN-weighted cosine_profile (global argmax), which drifts onto interferents
in narrow windows. The narrow-recovery + AUC metrics here exist to prove that
fix when it lands.
Validation
cycle, roc_auc, discrimination).
the suite base() closures; the new suites follow it for consistency.