Out-of-the-box auto-configuration (PR1: interpreter + validator + calibrator)#779
Draft
DLWoodruff wants to merge 34 commits into
Draft
Out-of-the-box auto-configuration (PR1: interpreter + validator + calibrator)#779DLWoodruff wants to merge 34 commits into
DLWoodruff wants to merge 34 commits into
Conversation
…ion) Requirements-only design doc for a --out-of-the-box mode that introspects the environment and model to auto-assemble a sensible mpi-sppy run. Captures the confirmed requirements (user options win, env/model probing, 3-rank floor with EF fallback, transparency via equivalent command line + improvement advisory, proper bundling) and parks the decision-logic mechanism (expert system vs neural net vs nested ifs, guided by dated data files) as the explicit open first design question. No library code yet. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Records the resolved decision-logic mechanism (authored declarative policy file + thin Python interpreter; no rules-engine library, no neural net) in the design doc (§5/§5.1) and ships the first dated policy file mpisppy/generic/ootb_policies/ootb_policy_2026-06-28.json. The policy is pure data with _comment fields throughout; every threshold is flagged _cold_start_guess pending benchmark data. No code consumes it yet. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Resolves the instantiation question as a user-selectable effort axis: --out-of-the-box-minus (no instantiation), --out-of-the-box (one probe scenario; default), --out-of-the-box-plus (all + brief solve; later). One interpreter + one policy; tiers differ only in gather_facts depth, and decisions degrade gracefully to advisories when a fact is absent. Adds the "probe not reuse-all" intervention-seam rationale and a probe_scenarios knob (default 1, lands with the base tier). Marks the bundling heuristic resolved, records PR1 (pipeline + minus + base) / PR2 (plus) phasing, and reconciles the §1/§2/§3 surfaces that previously said the instantiation question was TBD. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Reconcile surfaces left stale by the §5/§5.2 resolutions: - Status header no longer calls the mechanism "an open question" or claims "no library code yet" (sketch exists; policy committed). - Vocabulary: "single CLI switch" -> the three effort-tier flags. - Goal 1 + §5.1 agree: generic_cylinders.py is the CLI entry, OOTB code lives in the refactored mpisppy/generic/ package (was contradictory). - Output 4: EF gate is not only "< 3 ranks"; also too-few/too-large scenarios. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Each effort flag takes an optional policy-file path (domain=str, default=None,
argparse_args={'nargs':'?','const':<default>}): absent -> off, bare -> shipped
default (newest dated file with no focus token), PATH -> that file. Drops the
separate --ootb-policy-file flag, focus-name resolution, and any machine-read
focus field; multiple foci ship as differently-named files and the filename
documents the focus for humans. Notes the bool-domain caveat (pyomo forces
store_true for bool, so optional values require a str domain; add_to_config
forwards argparse_args to add_argument).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The "to improve" output is relabelled "Suggestions" and is written after the algorithm executes (so it can also reflect how the run went), with the equivalent command line still printed up front. Renames the policy key advisories -> suggestions and reconciles the design doc (goals, outputs, schema, tiers, phasing). Interpreter sketch (uncommitted) follows suit: make_suggestions(decision, facts, policy, outcome=None) is a post-run step, recommend() no longer computes them. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Bundle size becomes a derived quantity, not a specified one: pick the largest scenarios_per_bundle that divides num_scens, leaves >= B_min bundles, and keeps a bundle's modeled solve effort within budget. Adds a shared effort_scaling shape (continuous ~linear, integers superlinear via int_exponent, integer nonants a fixed coupling cost) and a bundle_sizing block. Two anchors on one shape: base uses a relative budget (effort vs a single scenario, unit-free), plus uses the same shape calibrated by a measured single-scenario solve time; minus falls back to a count target. New §5.3 documents it, including why measurement (plus) calibrates but does not remove the JSON shape (MIP times are noisy), and why the EF gate needs an absolute ceiling (so it stays count-based for now). Replaces the old bundling/target_bundles heuristic. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A general driver flag (plain store_true, not OOTB-specific): do the inspection, print the configuration + equivalent command line + config-time suggestions, then stop before the production run. Semantics are "no production run," not "no solver call ever": per the chosen resolution B, out-of-the-box-plus's brief calibration solves count as inspection, so plus + --inspect-only still measures then stops. Used standalone (no --out-of-the-box) it verifies a scenario can be instantiated, reusing the OOTB probe via a shared verify_instantiation. With --out-of-the-box-minus it is silly but allowed, with --inspect-only taking priority (one verification instantiation; decision stays minus-level). New §5.4; folded into PR1 scope. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Two refinements. (1) A policy additional_options list lets OOTB turn on extra command-line options beyond the structural choices (e.g. --max-iterations, or --grad-rho in a quality focus), applied via the user-options-win rule and skipped on the EF path; conditionality is expressed by which focus file ships the option, not by predicates. This subsumes param_defaults. (2) Suggestions are now mostly computed: small Python generators build messages from live facts/decision/run-outcome, so the prose lives in code and the policy only toggles/tunes them (suggestions.disabled). Decisions stay data-driven; suggestions are the computed diagnostics layer. Reworded the now-stale "named predicates" line in sec.5 to "plain coded checks". Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
With no size profile there is no safe way to size a bundle, so minus always runs unbundled (bundling requires base or plus). Drops the count-based minus fallback: removes fallback_bundles_per_intra_rank from the policy and the _pick_spb_by_count helper from the sketch; recommend() now only sizes bundles when a probe profile is present. Reconciles §5.1/§5.2/§5.3/§6. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The EF is all scenarios as one model, so the EF gate now reuses effort() on the whole problem (effort(num_scens)) against an absolute EF budget, resolving the previously-open size-aware EF gate. Above the rank floor: base runs EF when effort(num_scens) <= ef_effort_budget; plus would use ef_target_seconds via a measured t1; minus (no profile) keeps the count rule ef_if_num_scens_at_most. The EF budget is absolute (the monolith has no single-scenario reference) but shares effort units with bundle sizing, so the two budgets are mutually consistent. Adds ef_effort_budget / ef_target_seconds to the policy; updates §5.1/§5.3/§6. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Splits the extra (non-structural) options into per-concern option_categories
plus a catch-all additional_options. Each category block is {flag, value,
superseded_by}, where superseded_by lists the user flags that obviate OOTB's
default for that concern -- so OOTB backs off when the user addresses the
concern with ANY equivalent flag, not just the identical one. This is required
for correctness, not just tidiness: mpi-sppy allows only one rho setter active,
so adding --grad-rho when the user chose --sensi-rho would be a hard error;
rho_setter.superseded_by lists all rho setters. v1 categories: rho_setter
(--grad-rho), termination (--rel-gap 0.01), max_iterations (--max-iterations
100), dynamic_rho (--dynamic-rho-primal-crit, boolean; threshold defaults 0.1).
Catch-all entries may carry their own superseded_by (default = own flag).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…on TODO Two design items. (1) If the user explicitly set any decomposition flag (a wired spoke or non-default hub) and has at least the rank floor, OOTB never substitutes the EF, even for a small problem (requirement 0); the rank floor is still checked first. The decomposition-flag vocabulary is a fact about generic_cylinders, so it lives in the interpreter (DECOMPOSITION_FLAGS), not the policy. (2) Adds a Validation section (sec.8) sketching a fully automated policy-file validator that uses the examples: static schema checks plus behavioral checks (EF invoked when it should be, forced decomposition wins, bundling validity, no conflicting options, and round-trip executability of the emitted command line). Marked design TODO; checklist is partial. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…tios Three related changes. (1) max_cylinders 6 -> 7 so all six ladder rungs are reachable (no dead rung). (2) Roster sizing redesigned: prefer a small core (hub + lagrangian + xhatshuffle) widened by ranks, adding a further ladder rung only while each cylinder keeps >= min_ranks_per_cylinder ranks -- so 6 ranks gives 3 cylinders, not 6 single-rank cylinders. (3) Unbalanced rank distribution (flex-ranks): ranks split across chosen cylinders by per-cylinder rank_ratios (xhatter 0.2, default 1.0), normalized and floored at 1 each -- explicitly a crude cold-start, since the real split depends on subproblem solve cost (a plus-tier refinement). Adds §5.5; updates schema row and policy rank_allocation. The widest cylinder's rank count governs the bundling floor. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
--inspect-only N plans as if N ranks were available, so a supercomputer user can get the recommended command line for a large job from a login node without launching it; bare --inspect-only uses the detected ranks. Only the rank count is hypothetical -- solvers and model size still come from the real session -- and the emitted "mpiexec -np N ..." reflects it. This makes --inspect-only an optional-value flag (domain=str, nargs='?'), not store_true (same bool-domain caveat as the effort flags). Updates sec.5.4. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Expands the validation design (sec.8) into three layers fast-to-slow: static schema checks, fast decision checks (recommend() only), and a slower run tier that actually executes on the small examples. The run tier forces cylinders even where the EF gate would pick EF, runs them to near convergence, and verifies the behavior is about what we expect by comparing to the EF ground truth (objective/first-stage within tolerance, bounds don't cross) -- feasible because the small examples are EF-solvable. These need longer runs, so the run tier is nightly/opt-in while the fast tiers gate every change. Adds a report requirement: pass/fail per check with details, machine- and human-readable, naming the policy file and version. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
CI runners typically have no commercial solver and tight time budgets, so the per-PR gate is limited to the cheap, solver-free checks: layer 1 (static schema) plus the synthetic-facts subset of layer 2 (pure recommend() on hand-built Facts, no instantiation or solver). Example instantiation, anything needing a solver, and all of layer 3 (real runs) execute nightly / on demand / locally, not as a gate. Corrects the earlier "fast tiers in CI" overstatement in sec.8. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…terations
The report details every test (not just failures) and, for layer-3 runs,
records the iteration count and convergence status. It prominently highlights
runs that hit the iteration cap without converging ("maxed out on iterations")
-- a yellow flag that the recommended config or the policy's defaults converge
poorly on that example, even when the run technically completed. Updates sec.8.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…ts shape The validator is no longer a vague TODO: a dated policy file should not ship without the tool that validates it, so it lands in PR1. Nails down the concrete shape -- a runnable mpisppy/generic/ootb_validate.py over a fixed small example set (farmer, a small MIP, aircond), with the CI gate a pytest of the static + synthetic-facts layers wired into run_coverage.bash and test_pr_and_main.yml, and layer 3 run via --run nightly/locally. Adds the validator to PR1 scope in sec.7 and reworded the heading to "validator" (the tool), not "validation". Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…y-it path The validator is also the easiest way to try OOTB out: running it on the bundled examples exercises the whole pipeline end-to-end (decisions, real decomposition runs, EF comparison, the report), so it doubles as the demonstration harness. That means the full tool -- all three layers, including the run-tier -- ships in PR1, not a fast-follow; the small example set keeps the run-tier feasible with a modest/free solver. Records the rationale in sec.8. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…icy JSON) Audited the design doc against itself and the committed policy file. The schema table is an exact set match with the JSON config keys. Fixed four internal drifts: the status header now lists all resolved sections (not just §5/§5.2); sec.4 cites the EF gate at §5.3 (was §5.2); dropped the stale "~100 lines" interpreter-size claim; and sec.8 layer 2 now introduces its synthetic-facts (CI-gating) vs real-example (out-of-CI) split so the later CI note has a clear referent. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…sults The validator is not a correctness oracle -- we can't reasonably include expected results. Instead it runs the recommended configs, records what happened, and automatically flags two failure modes: EF recommended but not at a 1% MIP gap within ten minutes, and decomposition that terminates on max_iterations (including forced cylinders on small problems). Everything else is recorded for a human reader, who judges whether performance is acceptable -- which is why the report must be human-readable. Drops the EF-ground-truth / expected-vs-actual framing from sec.8. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A third tool (after the interpreter and validator), data-tuning side and not PR1: it fits the effort_scaling coefficients from timed solves across a benchmark spread so modeled effort tracks measured time, and produces an effort->seconds scale so the abstract budgets (ef_effort_budget, the base M) can be set and read in seconds (matching the existing plus_*_seconds budgets). Its output is updated coefficients + scale for a new dated policy file -- the producer side of the sec.5 migration path. sec.6 now names it as the dated-file generation mechanism (corpus/versioning cadence still open). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…ts output) A policy whose effort coefficients and budgets are hand-guesses is unassessable -- reviewers would be reviewing noise. So the effort-calibration tool ships in PR1, and the v1 policy's effort numbers are its output: an initial calibration on the example set (sharing the validator run-tier's timed solves), reproducible by re-running it; a broader benchmark corpus is the future refinement. Updates sec.9 status, the PR1 scope (sec.7), the schema note (sec.5.1), sec.6, and the status header. PR1 is now interpreter + validator + calibrator -- large but cohesive: together they make the shipped policy runnable and assessable. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
plus is still OOTB -- the same one-shot decisions as base, just with a richer fact base (measured solve time, LP/integrality gap). It does not iterate, search a parameter space, or auto-tune to optimality; it's "out-of-the-box with more information," not an autotuner. Noted in sec.5.2 and tied to the sec.1 non-goal; distinguished from the separate offline calibration tool (sec.9). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…e, wiring stubbed) Baseline for PR1 coding. The pure (facts, policy) -> Decision logic is complete and smoke-tested: recommend() (solver -> EF gate w/ effort budget + forced-decomp override -> small-core/widen spoke roster -> unbalanced flex-rank split -> effort-budgeted bundle sizing -> per-concern option_categories w/ superseded_by), _effort / _pick_spb_by_effort, _allocate_ranks, computed suggestion generators, load_policy (newest dated), command_line, DECOMPOSITION_FLAGS. Stubbed (TODO, the wiring): gather_facts probing (_detect_num_scens, size profile, verify_instantiation, user_flags), apply Decision onto Config, the run, and the plus measure-and-scale hook. ruff clean; imports clean. See doc/designs/out_of_the_box_design.md. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The interpreter sketch is committed (decision logic complete, wiring stubbed), so the status line no longer calls it uncommitted; next step is turning it into PR1 code. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…-end Turn the OOTB sketch into working code. The pure recommend(facts, policy) logic was already complete; this commit adds the environment/model wiring and hooks it into the generic_cylinders driver, so --out-of-the-box-minus and --out-of-the-box (base) now actually configure and run a model. - config.ootb_args(): the three mutually-exclusive tier flags plus --inspect-only, each an optional-value flag (str + nargs='?', const='') so a bare flag uses the shipped default policy and a value names a policy file. Wired into generic/parsing.py. - out_of_the_box.py wiring: effort/policy resolution, gather_facts (structural facts always; a one-scenario probe size profile at base/plus), verify_instantiation (the shared probe / model smoke-test), apply_decision (Decision -> cfg so the normal driver path runs it), and the configure/report_suggestions entry points (rank-0 printing). - generic_cylinders.py: run configure() right after arg parsing (mutating cfg), short-circuit on --inspect-only, and print the post-run Suggestions. Solver-key split (raised as a design issue): the EF path reads cfg.EF_solver_name and the decomposition path cfg.solver_name. recommend() now settles the solver NAME first and emits the path-appropriate flag once the EF gate decides; a naive user's --solver-name is carried over to --EF-solver-name when OOTB runs the EF (belt-and-suspenders guarantee in apply_decision too). Rank allocation now emits the per-spoke --<spoke>-rank-ratio flags and mirrors WheelSpinner.apportion_ranks for the bundling #bundles>=intra_ranks floor. Policy: add a base_rho option category (--default-rho 1). PH needs a base rho; --grad-rho refines but does not supply one and (unlike sep/coeff/sensi) is not auto-defaulted, so a --grad-rho decomposition without --default-rho was a hard error. Verified on farmer: EF (few ranks / small effort / few scens), decomposition (minus + base), proper bundling (base, forced-decomp), --inspect-only, and the solver-name->EF mapping all run. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add mpisppy/generic/ootb_validate.py -- a runnable validator that checks a
policy file is well-formed and that its recommended configurations make sense,
and (on demand) that they actually run, using the mpi-sppy examples as test
models. Three layers:
1. STATIC (schema): required keys/types; every referenced flag is a real CLI
option; spoke rungs, rho setters, _cold_start_guess keys, and
DECOMPOSITION_FLAGS all line up with the real generic_cylinders vocabulary.
2. DECISION (recommend() only, no solves): synthetic hand-built Facts (the
CI-gating subset) asserting EF-when-it-should, decompose-when-it-should,
forced-decomp-wins, bundling validity, and no conflicting rho setters; with
--examples, also probe-instantiated real models (farmer / sizes / aircond).
3. RUN (--run; slow, needs a solver + mpiexec, never a CI gate): actually run
the recommended EF and forced-decomposition configs and FLAG (a) an EF that
misses a 1% gap in ten minutes and (b) cylinders that max out on iterations.
Everything else (objective, gap, iters, walltime) is recorded for a human.
Produces a human-readable + machine-readable (--json) report naming the policy
file and policy_version.
CI gate: mpisppy/tests/test_ootb_validate.py runs only layers 1 + 2-synthetic on
every shipped policy (solver-free, ~0.5s), plus negative tests proving the
validator actually fails on malformed policies. Wired into run_coverage.bash and
test_pr_and_main.yml in the same commit.
Supporting changes:
- parsing.add_driver_args(cfg, m=None): the arg declaration split out of
parse_args so the validator can get the authoritative valid-flag set without
parsing a command line (m=None skips the model-specific adder).
- ootb_validate scrubs OMPI_/PMI_/... from child mpiexec environments: importing
mpi-sppy makes the validator a singleton MPI process, and leaving those vars in
the child made a fresh mpiexec abort instantly.
- Policy fix the validator itself caught: spoke_ladder._cold_start_guess listed
prose ("ladder ordering of priorities 3-6"), not a real key; moved that note
into the _comment so _cold_start_guess names only real keys.
Verified: layers 1+2 PASS on the shipped policy (50 static + 8 decision);
--examples PASS across all three models; --run completes all six runs (farmer /
sizes / aircond, EF + forced-decompose) with none flagged.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add mpisppy/generic/ootb_calibrate.py -- turns the abstract effort_scaling
coefficients and EF budget into data-tuned, interpretable values from timed
solves on the examples (the producer side of the dated-policy migration path).
1. Times extensive-form solves over a spread of bundle sizes across models of
differing continuous / integer / nonant content (farmer / sizes / aircond,
with calibration-tuned scenario counts so several bundle sizes are
reachable).
2. Fits cont_coeff / int_weight / int_exponent / int_nonant_coeff by
non-negative least squares per candidate integer exponent, choosing the
exponent by R^2.
3. Calibrates effort to ~SECONDS: the fitted coefficients are kept in seconds
units, so modeled effort approximates predicted solve time and the absolute
EF budget reads as roughly seconds (ef_effort_budget set from
ef_target_seconds) instead of an opaque large number. Significant-figure
rounding preserves the legitimately tiny coefficients (int_weight ~ 1e-9
when the exponent is 3 and there are ~150 integers -- decimal rounding would
zero it).
4. Writes a new dated policy with the fitted numbers and provenance in place of
the cold-start guesses.
CI test mpisppy/tests/test_ootb_calibrate.py covers only the PURE parts
(solver-free): coefficient recovery from synthetic points, the minimum-points
guard, significant-figure rounding, and that a fitted policy still passes the
validator's static + synthetic-decision layers. Wired into run_coverage.bash and
test_pr_and_main.yml. The measurement tier needs a solver and runs on demand /
locally, like the validator's run tier.
Verified on this box (gurobi): R^2 ~= 0.9995 over 14 timed solves; the resulting
effort model keeps a continuous problem on the EF to ~370k scenarios but
decomposes a 150-integer MIP after ~28 -- the MIP superlinearity is now real --
and the calibrated policy passes the validator.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Regenerate the shipped policy with ootb_calibrate (gurobi, R^2~=0.9995 over 14
timed EF solves on the example set), so the v1 numbers are the calibrator's
output rather than hand-guesses (design sec. 9):
- effort_scaling coefficients are data-tuned and in ~seconds units;
- ef_effort_budget is 120 (derived from ef_target_seconds), reading as roughly
seconds instead of the old opaque 100000;
- provenance + an effort_scaling._calibration block record the solver, R^2,
point count, and the per-machine/solver caveat.
These numbers are this reference machine's; the calibration is reproducible by
re-running the tool, and a later dated file folds in a broader corpus.
Also make the calibrator reconcile the prose it supersedes: the effort_scaling
and ef_fallback _comment fields no longer claim "All cold-start guesses" for the
now-calibrated coefficients / budget.
Fix test_ootb_validate's cold_start_guess regression test to inject into
bundle_sizing (which stays hand-authored) instead of effort_scaling (whose
_cold_start_guess the calibrator now removes).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Document OOTB as the recommended on-ramp (requirement 1). - New doc/src/out_of_the_box.rst: spirit, basic usage, what OOTB decides (solver, EF-vs-decompose, widened spoke core, flex-rank split, bundling, extra defaults), transparency (equivalent command line + Suggestions), the effort tiers and the seconds-calibrated EF gate, --inspect-only (incl. the assumed rank count for HPC planning), policy files and selection, and the validator and calibrator tools. - quick_start.rst: add "Recommended first run: let mpi-sppy configure itself" at the top of the farmer section, pointing to the full page. - generic_cylinders.rst: a tip box pointing newcomers at --out-of-the-box. - index.rst: add the page to the "Running with generic_cylinders" toctree. The examples are honest about the calibrated policy: on a fast machine with a commercial solver the small bundled examples solve as the EF within budget, so plain --out-of-the-box correctly chooses the EF for them; the note shows how to exercise the cylinder path (request a spoke, or use --out-of-the-box-minus). --out-of-the-box-plus is documented as reserved (currently behaves like base). Docs build clean (no new warnings). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #779 +/- ##
==========================================
+ Coverage 75.00% 75.68% +0.68%
==========================================
Files 166 169 +3
Lines 21581 22321 +740
==========================================
+ Hits 16186 16894 +708
- Misses 5395 5427 +32 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
The OOTB modules were ~54% patch coverage because large parts only run with a solver / MPI / a subprocess, which CI does not have. Close the gap two ways, without pretending to test what cannot run in CI: - New mpisppy/tests/test_out_of_the_box.py drives the interpreter wiring (gather_facts probe -> recommend -> apply_decision -> configure) on farmer with --inspect-only N for a deterministic rank count. Building a scenario does not solve, so the whole EF / decomposition / bundling decision-and-apply path runs solver-free, plus direct tests of apply_decision, command_line, the suggestion generators, requested/effort_and_policy, and _user_flags. - Extend the validator/calibrator tests to cover the remaining pure logic: format_report, _parse_decompose (both branches), main (pass/fail), _scen_cli, _child_env, example_models, validate_decisions_examples (scenario build, no solve), and _calibration_specs / _design_columns. - # pragma: no cover on the genuinely un-CI-able code: the validator run tier (validate_runs, _run_one — subprocess + solver), the calibrator measurement (_pick_solver, _time_ef_solve, measure_example, collect_points, run_calibration), the __main__ guards, and the generic_cylinders OOTB CLI hook (configure() itself is unit-tested). Coverage of the new modules: ootb_calibrate 100%, ootb_validate 95%, out_of_the_box 94%. test_out_of_the_box wired into run_coverage.bash and test_pr_and_main.yml. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…NLP/MINLP) The base/plus probe now classifies the model from its integrality (vars_int) and the max objective/constraint degree (new model_degree fact: linear / quadratic / more-than-quadratic) into one of six problem classes, and OOTB picks the first available solver from that class's preference list. So a model that is more nonlinear than quadratic routes to a general NLP solver (ipopt), and an integer model never routes to a continuous-only solver. - policy: preference_order_by_class with one preferred-solver list per class; NLP -> ipopt, MINLP -> baron/scip/bonmin/couenne (rarely installed -> warn), continuous QP/NLP drop cbc/glpk, integer lists drop ipopt. preference_order stays the master superset (detection + minus-tier fallback); each class list is a subset of it. - interpreter: _model_degree() detector (polynomial_degree None or > 2 => nonlinear) wired into the probe size profile; Decision.problem_class; class-routed selection with a no-solver-for-this-class suggestion. - validator: static check that all six classes exist and each is a non-empty subset of preference_order; synthetic checks that each class routes correctly and a nonlinear model with only MIP solvers picks nothing (no wrong fallback). - tests + design doc updated. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Out-of-the-box auto-configuration — PR1
Draft for early review. This is PR1 of a phased rollout of the
--out-of-the-box(OOTB) mode described indoc/designs/out_of_the_box_design.md. It lets a relatively new user obtain adefensible mpi-sppy run with almost no knowledge of the library's internals:
supply a model module and scenario data, add
--out-of-the-box, and the driverintrospects the environment + model and assembles a configuration (solver, EF vs.
decomposition, spokes, flexible rank split, proper bundling), prints the
equivalent explicit command line, runs it, and prints a Suggestions list.
User-supplied options always win — OOTB only fills gaps.
What's in this PR
mpisppy/generic/out_of_the_box.py— purerecommend(facts, policy) → Decision, plusgather_facts,apply_decision, and theconfigure/report_suggestionsdriver hooksconfig.ootb_args()— the three tier flags +--inspect-only, as optional-value flags (str+nargs='?'); wired throughgeneric/parsing.pyandgeneric_cylinders.pympisppy/generic/ootb_policies/ootb_policy_2026-06-28.json— dated, declarative knowledge (solver order, EF budget, spoke ladder, rank ratios, effort model, extra-option defaults)mpisppy/generic/ootb_validate.py— static schema + decision checks + an opt-in--runtier that flags problem runs (design §8)mpisppy/generic/ootb_calibrate.py— fits the effort coefficients from timed example solves; its output is the shipped policy's numbers (design §9)doc/src/out_of_the_box.rst+ quick-start on-ramp +generic_cylinderstipEffort tiers
--out-of-the-box-minus— instantiate nothing (structural decisions only; cannot bundle)--out-of-the-box(default) — one probe scenario → size-aware EF gate and bundling--out-of-the-box-plus— reserved for PR2 (instantiate all + a brief timed solve); currently behaves like baseSeconds-calibrated effort
The effort coefficients are produced by the calibration tool from timed EF solves
on the examples and kept in seconds units, so the EF budget reads as a
wall-clock target (
ef_effort_budget = 120, i.e. ~120 s) rather than an opaquenumber. The shipped numbers are this reference machine's (gurobi); they are
per-machine/solver and reproducible by re-running the calibrator. The resulting
gate is sensible: a continuous problem stays on the EF to a large scenario count,
but an integer-heavy MIP decomposes far sooner (the superlinearity is real).
CI vs. local
The per-PR gate is solver-free and fast: the validator's static layer plus
the synthetic-facts decision layer, on every shipped policy, via
mpisppy/tests/test_ootb_validate.py; and the calibrator's pure fit viampisppy/tests/test_ootb_calibrate.py. Both are wired intorun_coverage.bashand
test_pr_and_main.yml. Everything that needs a solver — exampleinstantiation, the validator's
--runtier, and the calibrator's measurement —runs locally / nightly, not in CI.
Try it
Notes for reviewers
plustier is intentionally out of scope (PR2).gather_factsalreadytreats base/plus the same single probe, so the only PR2 delta is the extra
measurement.
rank-ratio cold-start choices, and on whether shipping a machine-calibrated
policy in the repo is the right call (the alternative is to ship the tool and
calibrate per environment).
🤖 Generated with Claude Code