UK: household-wealth imputation stage with cash / S&S ISA split#181
UK: household-wealth imputation stage with cash / S&S ISA split#181vahid-ahmadi wants to merge 2 commits into
Conversation
Declare a WAS-donor `household_wealth` source stage (manifest + plan module, mirroring the bus imputation pattern) that surfaces `cash_isa` and `stocks_and_shares_isa` as standalone outputs, with the investment-ISA component also folded into `corporate_wealth` for back-compatibility. Addresses the gap from the archived UK data wealth imputation, where investment ISAs were summed into corporate_wealth (not separable) and cash ISAs were never represented. The matching model-side input variables are in PolicyEngine/policyengine-uk#1791. Draft: the executable WAS donor transform is injected by the build runtime (not in this manifest by design); requesting review on stage placement given the UK build module (bus, geography) currently lives on uk-bus-fare-imputation-calibration. Closes #180 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The uk/ build package is a spec-only country package (JSON resources listed in country_package.json; runtime Python lives in shared modules). The first cut added a plan module and __init__.py into uk/, violating the spec-only contract. - Drop uk/wealth_imputation.py and uk/__init__.py. - Register wealth_source_stages.json in uk/country_package.json resources. - Rewrite test_uk_wealth.py to load the manifest via the shared load_source_manifest runtime and assert the cash/S&S ISA split, the fit+clip operations, and the fold into corporate_wealth. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
MaxGhenis
left a comment
There was a problem hiding this comment.
Right direction, and the manifest passes the real load_source_manifest validator (operation vocabulary, nonnegative_outputs ⊆ outputs with net_financial_wealth correctly excluded, no incumbent deps) with meaningful tests. A few things before this is mergeable (agent-assisted review):
-
The description is stale. It describes a Python
uk/wealth_imputation.pyplan module mirroringus_plan/uk_bus_planand a stacking concern against the bus-fare branch. The head commit made this spec-only — the actual diff iswealth_source_stages.json+ acountry_package.jsonresource line + contract tests, it applies cleanly to main, and touches no__init__.py. Please rewrite the body to match what's here. -
As merged this is inert — nothing loads it.
us_runtimewiresUS_SOURCE_MANIFEST = load_source_manifest(...)andus_planrequires a transform per stage;uk_runtimehas no equivalent and doesn't referencesource_manifest,source_stages, or wealth at all. So this declares the contract but produces nocash_isa/stocks_and_shares_isacolumns until a follow-up wiresuk_runtime. Fine as a first step — just say so and link the follow-up. -
Naming/placement (you asked): US keeps one
source_stages.jsonper country with many stages. I'd match that —uk/source_stages.jsonwith ahousehold_wealthstage — so the eventual UK manifest loader mirrors the US one and bus/geography stages append rather than spawning more*_source_stages.jsonfiles. -
No calibration. It's a donor QRF clipped to donor range, no target. HMRC's 2022-23 adult ISA market is ~£726bn (S&S ~£430bn, cash ~£294bn) and WAS under-captures the top tail, so a raw WAS-donor impute won't hit those totals. Either add a calibration target downstream or document that ISA totals are uncalibrated.
-
Minor: the WAS mapping (
cash_isa ← DVCISAVR8,stocks_and_shares_isa ← DVIISAVR8_aggr) lives only innotesand isn't test-checked — worth a line confirming DVCISAVR8 is the cash-ISA value field.fit_weighted_qrfhas no seed; set one for reproducible builds. And the body saysDVIISAVR8while the manifest saysDVIISAVR8_aggr.
No pin issue with the model PR — the manifest is just column names, so nothing here depends on policyengine-uk#1791 at build time. But the runtime PR that actually emits these columns should bump the uk extra from policyengine-uk>=2.88 to >=2.90 (the release carrying the input variables) so the columns land instead of being dropped.
What
Adds a WAS-donor
household_wealthsource stage — a declarative manifest (uk/wealth_source_stages.json) plus a plan module (uk/wealth_imputation.py) mirroringuk_bus_plan/us_plan— that imputes UK household wealth holdings and surfacescash_isaandstocks_and_shares_isaas standalone outputs. The investment-ISA component is alsofold_intocorporate_wealthfor back-compatibility.Why
Closes #180. The archived UK-data wealth imputation summed investment ISAs into
corporate_wealth(so they weren't separable) and never represented cash ISAs, leaving ISA balances unusable. Model-side input variables added in PolicyEngine/policyengine-uk#1791.For review (@MaxGhenis)
uk/__init__.py, bus imputation, geography) currently lives onuk-bus-fare-imputation-calibration, notmain. I added a minimaluk/__init__.pyexporting only the wealth symbols; this will need reconciling/stacking with that branch when it lands — guidance welcome.notes:cash_isa←DVCISAVR8(not mapped in the legacy imputation),stocks_and_shares_isa←DVIISAVR8.test_uk_wealth.py(manifest validity, plan assembly + donor citations, ISA split present/nonnegative, fold-into corporate_wealth), mirroringtest_uk_bus.py. Passing locally.