Bootstrap/bagging confidence intervals (boot-sp merge PR-1: empirical core + schultz)#783
Open
DLWoodruff wants to merge 14 commits into
Open
Bootstrap/bagging confidence intervals (boot-sp merge PR-1: empirical core + schultz)#783DLWoodruff wants to merge 14 commits into
DLWoodruff wants to merge 14 commits into
Conversation
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…irical core, run_all.py entry, real MPI tests) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Port the empirical (numpy-only) part of boot-sp into mpisppy/confidence_intervals/bootsp/: the classical, extended, subsampling, and bagging confidence-interval methods, the user and simulation drivers, and boot_general_prep. - scipy-free: statistics.NormalDist().inv_cdf replaces scipy.stats.norm.ppf, so the empirical core imports with no scipy present (and on scipy>=1.14). - MPI comms consolidated into boot_utils; the empirical-vs-smoothed and method dispatch is consolidated into boot_sp.compute_ci. - BootMethods enum ships complete (11 members); a Smoothed_* method raises a clear "not yet merged" error in both user_boot and simulate_boot. statdist and the smoothed methods land in PR-B. - compute_xhat looks for the fixed name xhat_generator first, falling back to the legacy xhat_generator_<module>. Example: examples/bootsp/schultz (unique + nonunique). schultz has fully deterministic integer data, so the EF optimum and bootstrap draws are solver- and rank-independent. Added a fixed-name xhat_generator and a uniform-probability fallback; self-contained JSONs (compute xhat/optimal live) and a schultz.bash demo. do_one_boot registers an np=2 schultz run in part 1 of examples/run_all.py. Tests: test_boot_sp.py (methods, enum, xhat, smoothed rejection, locked values) and test_boot_sp_simulate.py (coverage harness + real mpiexec -np 2 Gatherv value assertions, locked per comm size), wired into the confidence-intervals CI job and run_coverage.bash in this commit. Two fixes exposed by the real-MPI path: the method functions returned a 3-tuple of None on non-root ranks while the drivers unpack 6 values (now 6 Nones), and cfg_from_json's _dobool could not set the outer badtrip flag (added nonlocal). Doc: doc/src/boot_sp.rst (+ index toctree). Design doc reconciled to the schultz-only PR-A scope (bootsp_merge_design.md sections 2.5/3/5/6/8). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…orkflow Record the intended third-PR shape: given a dataset, generic_cylinders finds xhat from part of the data and a bootstrap confidence interval from the rest (a hold-out split mapping onto candidate_sample_size/sample_size). Does not change the two-PR plan for the merge itself. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
A companion to the on-the-fly schultz example that reads its scenario data from a committed dataset instead of computing it, to document the data-based bootstrap workflow (and as the eventual demo for the generic_cylinders data-split follow-on). - examples/bootsp/schultz_data/: schultz_data.py (same model; scenario_creator reads row scennum from schultz_data.csv, cached, path resolved relative to the module; --data-file option; xhat_generator threads data_file), a seeded generator (schultz_data_generator.py) and its output schultz_data.csv (200 sampled (xi1, xi2) observations), a self-contained json, and a demo bash. Still statdist-free and scipy-free. - test_boot_sp.py: a Test_boot_sp_data class (dataset reader incl. missing-file error, xhat, locked ci values, coverage) using the committed csv. - run_all.py: do_one_boot generalized to take dir/module/size args; adds an np=2 schultz_data run alongside schultz in part 1. - boot_sp.rst: a "Working from a dataset file" section showing the csv and the xhat-from-part / bootstrap-on-the-rest split. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…PR roadmap Extend the design per DLW's direction: - Elevate the ultimate goal to the top of the document: data-based bootstrap/bagging computed from a given dataset, available directly in generic_cylinders (the data-based analog of the MMW option). Added as goal 1 and as an early "Ultimate goal" callout; removed from non-goals. - Recast the effort as a stacked, multi-PR roadmap in two stages: Stage 1 moves boot-sp in (PR-1, PR-2, on the dependency boundary); Stage 2 integrates into generic_cylinders (PR-3, possibly PR-4). PRs stack (branch off the predecessor) for review/testing granularity. - New section 9 designs the integration: the --boot-* option group over a data file; the key scenario-name-vs-position reconciliation (scenario_creator stays name-based while bootstrap/bagging partition and resample by position in the canonical name list, via a unique sample-name -> position mapping that generalizes boot-sp's Scenario/SampleScenario trick); features that do not apply (MMW and sequential sampling are mutually exclusive with a fixed dataset; unbounded generation; multistage); and open questions. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…linders Resolve the §9.4 open question: in the generic_cylinders integration the candidate (M) and resampling-pool (N) records must not overlap. Recorded as a correctness requirement (a bootstrap CI on xhat's gap must be estimated on data that did not choose xhat, else it is in-sample/optimistic), stricter than boot-sp's standalone classical bootstrap. The positional layer makes it natural via disjoint position blocks. Moved out of open questions into a decided rule in §9.1. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…k tension Per DLW clarifications: - Dataset contract DECIDED: the model interprets the dataset (it may be a csv, a database, or a web resource, with no simple data->scenario map), so data loading and any data-source flag are the model's; the framework needs only the canonical scenario-name list and its length for the positional M/N split. - xhat solve DECIDED: uses the driver's configured solve. Captured the architecture: mirror generic/mmw.py -- a boot_args() group plus generic/boot.py::do_boot(module_fname, cfg, wheel=None) called after the main run, taking xhat from the wheel (or a file). - Rank usage (the real tension) expanded into regimes: small batches -> phase-separated flat Gatherv (natural first target); large batches -> nested MPI groups or serial cylinder solves. Left explicitly unresolved. - Flag names moved to their own section as a proposal to be finalized. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…scope Per DLW decisions: - Dataset size: no flag. The framework calls a model function that returns all scenario names implied by the dataset and derives the size from that list; documented that xhat-finding works on a subset (the M candidate records). - M is an explicit --boot-candidate-sample-size; mutually exclusive with --boot-xhat-input-file-name (omit or 0 when an xhat file is supplied). - Finalized the --boot-* flag set (no data-source/dataset-size flags) and noted boot_requested()'s validation. - Batch ranks: the first integration assumes a batch EF is solvable on its own (phase-separated flat Gatherv); the large-batch nested-rank case is a scheduled enhancement to be taken up right after the first PR stack merges (reflected in the §6 roadmap). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Fold in the SLURM constraint and the large-single-scenario reality: - SLURM -> in-process nested MPI groups (Comm_split within one allocation), not job-level sbatch orchestration. - Large single scenarios make decomposed batches a first-class case, so unify on "a batch is solved by a group of K ranks": K=1 is a per-rank EF (ships first), K>1 runs a wheel on the group's sub-communicator (same code path). R ranks split into G groups of K; nB batches distributed and Gatherv'd. - One singular batch sub-config (not per-batch, not boot_*-prefixed twins); for K=1 it's the boot solver role, for K>1 a nested generic_cylinders arg set. - Correct the earlier mischaracterization: the wheel already runs on a passed-in sub-communicator (WheelSpinner.run(comm_world=...), SPBase mpicomm); the only residue is two logging-only COMM_WORLD module globals tracked in Pyomo#782, so K>1 is largely unblocked. - §9.5: add --boot-batch-ranks (K) and the boot solver role (--boot-solver-name/-options via solver_specification(["boot",""])), which disambiguates the batch solve from the xhat solve (incl. the xhat-is-also-EF case). §6 roadmap updated to the K=1 / K>1 vocabulary. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
… flag A careful reader asked "what will the other ranks be doing?" — the doc didn't say. Fix: - §9.4: state plainly that the bootstrap phase partitions ALL R ranks into G = R // K concurrent groups (only R mod K leftover), no idle ranks; K is the group size (ranks cooperating on one batch), and the two phases are temporal over the same ranks. Trimmed the now-duplicated phase sentence. - §9.5: rename --boot-batch-ranks -> --boot-ranks-per-batch (the old name reads like "ranks assigned to batches", implying a subset) and spell out that it is a partition, not a carve-out. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
… claim The _sub_resample tron debug line (commented out in boot-sp, made live in the port) called evaluate_scenarios with stale arguments and would raise TypeError when tron is on; print the already-computed evaluation instead. The dataset-file doc section claimed the candidate rows and the bootstrap pool are disjoint, but the standalone estimators draw the pool from the whole dataset; say so, and note the strict split arrives with the generic_cylinders integration. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## main #783 +/- ##
==========================================
+ Coverage 75.41% 75.44% +0.03%
==========================================
Files 167 172 +5
Lines 21946 22533 +587
==========================================
+ Hits 16550 17000 +450
- Misses 5396 5533 +137 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
…e cap) test_data_file_coverage ran the coverage simulation, whose process_optimal solves the full 200-scenario schultz_data extensive form to get the reference z*. That single solve exceeds size-limited solvers on CI -- the community edition of cplex aborts with error 1016 (problem size limits exceeded) -- while the smaller resampled batches solve fine. Ship schultz_data_optimal.npy (the value process_optimal computes, with gap 0) and point the coverage test at it via optimal_fname, so the simulation reads the optimum instead of solving that one oversized EF. The stored value is exactly what the solve returns, so the coverage rate and length are unchanged. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This is PR-1 of the boot-sp merge (design:
doc/designs/bootsp_merge_design.md, included here): the empirical bootstrap/bagging confidence-interval core, its drivers, the statdist-free schultz examples, tests, and docs. PR-2 (the statdist distribution library + the smoothed methods + the remaining examples) follows once this lands. The end goal of the effort is data-based bootstrap/bagging CIs directly ingeneric_cylinders(design doc §9); this PR delivers the estimators as a library withuser_boot/simulate_bootentry points.Methods and software are described in Chen & Woodruff (IJOC 2023; CMS 2024).
What's here
mpisppy/confidence_intervals/bootsp/: the estimator engineboot_sp.py(classical gaussian/quantile, extended, subsampling, bagging with/without replacement),boot_utils.py(BootMethods enum, Config setup, module loading, xhat computation), and the driversuser_boot.py,simulate_boot.py,boot_general_prep.py— all runnable withpython -m.statistics.NormalDist().inv_cdfreplacesscipy.stats.norm.ppf).BootMethodsenum ships now; requesting aSmoothed_*method raises a clear "not yet merged — use the boot-sp package meanwhile" error until PR-2.examples/bootsp/schultz(scenario data computed from the scenario number; deterministic, solver-independent results) andexamples/bootsp/schultz_data(the same model reading a committed CSV dataset — the data-based workflow the methods are designed for). Both registered inexamples/run_all.py.mpisppy/tests/test_boot_sp.pyandtest_boot_sp_simulate.py, including realmpiexec -np 2tests with value assertions on the Gatherv batch parallelism (locked values keyed by comm size). Wired into the confidence-intervals CI job andrun_coverage.bashin this same PR.doc/src/boot_sp.rst, in the toctree after seqsamp.Notable reconciliations during the port (design doc §4)
xhat_generatoris looked up first, with the boot-sp legacyxhat_generator_<module>as a fallback.boot_utils/boot_sp.compute_ci).get_solver()/skipIf/round_pos_sigand__file__-relative paths instead of boot-sp's solver-digit and cwd assumptions.The farmer, cvar, and multi_knapsack examples build their scenario data with statdist distributions even on the empirical path, so they land in PR-2 with statdist — each example added exactly once (design doc §8).
🤖 Generated with Claude Code