Upgrade engine to 0.47 and observability to 1.3 as exact pins; drop unused pydantic-ai

[anth-volk]

Fixes #174
Fixes #173
Fixes #170

What

Migrates both capped dependencies to their current upstream versions, switches them to exact == pins, and removes the one dependency that blocked the resolve.

policyengine-uk-compiled >=0.44.0,<0.45 → ==0.47.0

The one breaking change in between: since 0.45, Simulation requires an explicit data source — data-less Simulation(year=...) raises ValueError.

• Parameter-lookup call sites (backend/tools/dispatch.py ×2, modal_app.py _preload_engine): pass dataset="frs". Construction is lazy and get_baseline_params() shells out to the binary, so no survey data loads on these paths. Can be simplified when Expose dataset-free baseline parameter export policyengine-uk-v2#112 (dataset-free parameter export) ships.
• Sandbox contract preserved (backend/engine/sandbox.py): SafeSimulation injects dataset="frs" when model-written code passes no data source — the run_python tool description and tool-contract eval cases document data-less construction, and the FRS guard bookkeeping already treated missing dataset as "frs". test_rejects_frs_row_level_microdata doubles as the regression test.
• Also lands with the jump: fiscal-year 2030/31 parameters, realterms module, DWP cost-of-living payments (2022/23–2023/24), mortgage interest in AHC, region-level LHA rates, local-authority EFRS, dataset version pinning. Baseline results shift for affected years/measures; no output-contract changes.

policyengine-observability >=1.0.0,<1.1 → ==1.3.1

• Since 1.1, structured logs route through a severity-aware destination manager: handled-error operation logs emit at WARNING instead of INFO (the reason for the original cap). The one log-capture test asserting an operation_failed record now captures the WARNING level.
• 1.3.1 pins the fixed OTel instrumentor (≥0.64b0), resolving the _IncludedRouter AttributeError that 500'd every CORS preflight on FastAPI ≥0.137 (Production outage: every CORS preflight (OPTIONS) returns 500 — OTel FastAPI instrumentation vs FastAPI >=0.137 #167). The instrument_fastapi=False workaround from Disable OTel FastAPI auto-instrumentation to stop OPTIONS preflight 500s #169 is removed; the CORS preflight regression test now exercises the instrumented app, and the instrumentation config test flips to assert it is enabled (closes Re-enable FastAPI OTel instrumentation once the observability pin ships and our constraint widens #170).
• Structured records gain severity and segment_tree fields; core fields now take precedence over caller attributes (no UK chat annotate() keys collide). This unblocks Route observability logs to Google Cloud Logging #142 (Google Cloud Logging destinations).

Exact pins + updater format change

Range pins let a new in-range release reach production on the next image build with no PR and no tests — the silent-drift failure class behind #167/#168. With == pins, every version change arrives as a tested PR via the dependency-update workflow. dependency_bump.py is updated in lockstep (same PR, so main never has a format the updater can't parse):

• same-minor releases → patch bump PR, auto-merge on green (was: in-cap floor bump);
• releases crossing a minor → bump PR left for review (was: cap widening);
• discovery still proposes the newest same-minor patch even when a newer cross-minor release exists.

Unit tests updated for the new format (20 cases); a live DRY_RUN=1 sweep against PyPI reports both pins current. This is a step toward #168 (full dependency locking) for the two first-party packages; transitive deps still float.

Removed: pydantic-ai[anthropic]

Nothing in the backend imports it (its only reference was a vestigial importorskip in a test whose imports don't need it), and docs/engineering/skills/uk-chat-runtime.md already states Pydantic AI is not the active orchestration layer. It also hard-blocked this upgrade: pydantic-ai pulls logfire, whose opentelemetry-sdk<1.43.0 cap conflicts with observability 1.3.1's >=1.43.0 floor, sending pip into deep backtracking (locally ending in a failed temporalio source build). Dropping it removes a large transitive tree (logfire, temporalio, xai-sdk, cohere, mistralai) from the production image. The anthropic SDK is a direct requirement and is unaffected.

Verification

Fresh Python 3.13 venv resolved from this branch's backend/requirements.txt (compiled 0.47.0, observability 1.3.1, otel-instrumentation-fastapi 0.64b0 — no resolver backtracking): 227 passed, 10 skipped, including compiled-integration tests, sandbox FRS-guard tests, the CORS preflight tests against the instrumented app, and the updated dependency-bump script tests. As a bonus validation of the original bug: an intermediate env with instrumentation enabled but the old 0.63b1 instrumentor reproduced the preflight 500s exactly.

Notes

• Once merged, the dependency-update workflow's pending proposals for both packages become no-ops; if the daily sweep opened chore: bump ... PRs in the old range format in the meantime, close them in favour of this one.
• Route observability logs to Google Cloud Logging #142 should be rebased on this.

🤖 Generated with Claude Code

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated (UTC)
policyengine-uk-chat	Ready	Preview, Comment	Jul 3, 2026 7:27pm

[github-actions]

Beta preview has been cleaned up because this PR was closed.