refactor(core): pi-ai migration spike — scope and path selection

[christso]

Refs #1205. Spike — not yet implementing the migration.

Status

This PR captures a finding the spike needs to resolve before any provider gets ported. The original plan in #1205 ("thin adapter behind `Provider.invoke()`") doesn't match the actual call graph.

What changed in this push

• `docs/plans/1205-pi-ai-spike.md` — full write-up of the call-graph finding and the two viable migration paths.
• `packages/core/src/evaluation/validation/targets-validator.ts` — fix pre-existing Biome import-order lint that was blocking push from any branch on this repo (introduced by feat(core): coerce interpolated env vars to native types in config.yaml #1203, somehow merged green).

The finding

Graders don't go through `provider.invoke()` — they call `provider.asLanguageModel()` and feed the returned Vercel `LanguageModel` into `generateText({ model, tools, ... })` themselves. Concrete sites:

• `graders/llm-grader.ts:485` — multi-step + filesystem tools
• `graders/llm-grader.ts:1106` — LLM-judge mode
• `graders/composite.ts:343`
• `generators/rubric-generator.ts:35`
• `providers/agentv-provider.ts:73` — `invoke()` actively throws

So the migration is coupled to the Vercel `LanguageModel` type, not to AgentV's `Provider.invoke` shape. Two paths:

Path A — Vercel LanguageModelV2 shim over pi-ai. Keep `asLanguageModel()`, return a shim that implements V2 over pi-ai's `complete()`/`stream()`. Zero consumer changes; non-trivial shim to maintain.

Path B — Drop `asLanguageModel`, enrich `Provider.invoke()` (tools, multi-step). Migrate the four consumers. Cleaner endpoint; bigger initial blast radius.

Plan doc has the full trade-off matrix.

Decision needed before next push

Pick A or B. My lean is A with a clear pivot threshold: if the V2 shim balloons (stream parts, tool-call deltas), abandon for B before merge.

Test plan

• Decide Path A vs Path B (blocking)
• Spike implementation: port one provider end-to-end on the chosen path
• Run grader-score baselines against an OpenAI-targeted eval (`scripts/check-grader-scores.ts`)
• Document remaining gaps (Anthropic thinking budget, Azure Responses parity, retry semantics)
• Update refactor(core): migrate LLM providers from Vercel AI SDK to @mariozechner/pi-ai #1205 with spike findings and a scoped follow-up plan for the full migration

Notes

• Plan doc lives in `docs/plans/` per AGENTS.md — to be deleted before the spike merges, with relevant content moved into refactor(core): migrate LLM providers from Vercel AI SDK to @mariozechner/pi-ai #1205 and module headers.
• The lint fix is unrelated to the spike but was required to push. Calling it out so reviewers don't expand scope expectations off it.

🤖 Generated with Claude Code

[cloudflare-workers-and-pages]

Deploying agentv with    Cloudflare Pages

Latest commit:	14564a2
Status:	✅  Deploy successful!
Preview URL:	https://face56d1.agentv.pages.dev
Branch Preview URL:	https://refactor-1205-pi-ai-spike.agentv.pages.dev

View logs

[christso]

Step 1 UAT — green

Ran a live end-to-end against OpenRouter (OpenAI-compatible endpoint) at commit 3eaca6ed on this branch:

const provider = resolveAndCreateProvider({
  name: 'openai-via-openrouter',
  provider: 'openai',                            // routes through new pi-ai-backed OpenAIProvider
  endpoint: 'https://openrouter.ai/api/v1',
  api_key: '${{ OPENROUTER_API_KEY }}',
  model: 'openai/gpt-4o-mini',
}, env);

const rubrics = await generateRubrics({
  criteria: 'A function that returns the sum of two numbers, handling integer overflow correctly.',
  provider,
});

Output:

provider.kind=openai, id=openai:openai-via-openrouter
got 6 rubrics:
  - id=sum_function_structure required=true weight=1 outcome="The function is correctly structured..."
  - id=input_handling required=true weight=1 outcome="The function accepts two integer inputs..."
  - id=integer_overflow_management required=true weight=2 outcome="The function correctly implements checks..."
  - id=correct_sum_calculation required=true weight=1.5 outcome="The function accurately calculates..."
  - id=edge_case_tests required=false weight=1.5 outcome="The function has been tested with edge cases..."
  - id=documentation_and_comments required=false weight=1 outcome="The function is well-documented..."

Confirms:

• OpenAIProvider.invoke() correctly routes through pi-ai's complete() (no ai-sdk generateText involvement on this path).
• rubric-generator.ts consuming provider.invoke() returns the same well-formed RubricItem[] shape it always did.
• pi-ai works against an arbitrary OpenAI-compatible endpoint (OpenRouter here) via just baseUrl + apiKey.

"Red" evidence is the test the unit-test mock change captures — the previous assertion that generateTextMock was called once for the openai target now fails under this branch (and is replaced by piCompleteMock / piGetModelMock assertions). See targets.test.ts:1186 change in 3eaca6ed.

[christso]

End-to-end UAT — orchestrator path through OpenAI provider

Ran the real agentv eval CLI command (not a SDK smoke test this time) against the new pi-ai-backed OpenAIProvider, routed through OpenRouter so we exercise the actual OpenAI-target code path:

OPENAI_ENDPOINT=https://openrouter.ai/api/v1 \
OPENAI_API_KEY=$OPENROUTER_API_KEY \
OPENAI_MODEL=openai/gpt-4o-mini \
LLM_TARGET=openai AGENT_TARGET=openai GRADER_TARGET=openai \
bun apps/cli/src/cli.ts eval examples/features/rubric/evals/dataset.eval.yaml \
  --output /tmp/spike-uat

Eval ran clean — 5/5 test cases, no orchestrator errors, mean score 78%, one expected quality_failure on code-quality-multi-eval (the test's code-grader returns 0 because there's no fixture script in the workspace — pre-existing).

Grader-score baseline (regression gate per AGENTS.md)

✓ code-quality-multi-eval   / rubrics: 0.5  ∈ [0.3, 1]
✓ code-explanation-simple   / rubrics: 1.0  ∈ [0.8, 1]
✓ technical-writing-detailed/ rubrics: 0.87 ∈ [0.8, 1]
3 passed, 0 failed (in this suite)

All three rubric-grader scores from the new pi-ai-backed OpenAI provider land in the ranges established before the migration. No drift.

JSONL output shape — no regression

Sampled all 5 result rows:

field	preserved?
scores[].type (llm-grader, code-grader, kebab-case)	✅
scores[].token_usage ({input, output} shape)	✅
execution_status (ok / quality_failure)	✅
costUsd field	✅ correctly absent (pi-ai's registry has no OpenRouter pricing → cost.total === 0 → field omitted, matching the prior Vercel path which also never set it)

No new fields appearing where there weren't any. No fields disappearing that should be present.

Evidence checklist (AGENTS.md "Completing Work — E2E Checklist")

• Preflight: .env present in worktree
• Unit tests pass (1740 core + 67 eval + 499 cli)
• Manual red/green UAT — green captured here; "red" is the existing targets.test.ts openai test rewritten in 3eaca6ed (would fail under this branch without the mock change, asserting the new pi-ai contract)
• No regressions in adjacent areas — Azure/OpenRouter/Anthropic/Gemini providers untouched, all 4 still go through the legacy invokeModel path
• Live eval verification — this run
• Grader-score baseline check — all 3 in range
• Studio UX verification — backend response shape unchanged so not strictly required, but worth a 5-min sanity check before merge

Now genuinely ready for review.