Unify Anthropic and OpenAI runtimes on pi-agent-core

[youknowriad]

Related issues

• Related to Add OpenAI/GPT model support to the Studio code agent #3244 — this is a direct follow-up to the OpenAI/GPT support PR. That branch added a parallel pi-based runtime alongside the Claude Agent SDK runtime; this PR collapses both into a single pi-based loop.

How AI was used in this PR

Substantial AI-generated migration. The plan, the runtime rewrite, the test updates, the documentation prose, and the commit message were all produced by Claude Opus 4.7 working from a detailed exploration spec. A human reviewed every file before commit, but the diff is large and AI-shaped.

The user explicitly opted for a direct cutover (no feature flag), so Studio runs every model — Claude Sonnet/Opus and GPT 5.5 — through the new unified runtime starting on first launch with this branch.

This is a Proof of Concept PR per the AGENTS.md guidance for substantial vibe-coded changes — opening it as draft + labeled `Proof of Concept` so the team can evaluate the architecture before considering merge. Expect rounds of refinement on cache hit rate parity, the wpcom auth path, and tuning the Claude-side skill appendix narrowing before this is merge-ready.

Proposed Changes

• Single agent loop. `apps/cli/ai/runtimes/pi/` is now the only runtime. `runtimes/anthropic/` is gone. `agent.ts` no longer dispatches by model family — `piRuntime` handles everything and resolves the family internally to pick between pi-ai's `anthropic-messages` and `openai-completions` providers.
• `@anthropic-ai/claude-agent-sdk` dropped. Removed from `apps/cli/package.json` dependencies. The forge build no longer needs the vendor-binary cleanup pass on Windows code-sign. The local `apps/cli/ai/sdk-message-types.ts` mirrors the SDKMessage and `SdkMcpToolDefinition` shapes the rest of the codebase consumes; the new `apps/cli/ai/tools/define-tool.ts` provides the `tool()` helper with the same authoring signature so all 24 tool files keep working unchanged.
• `studio mcp` standalone server. Re-implemented on top of the MCP SDK's `McpServer` directly (was using the agent SDK's `createSdkMcpServer` wrapper). Same tool catalog, same registration shape.
• Real result-message accounting. The synthetic `result` SDKMessage now carries `total_cost_usd` (summed from pi-ai's per-stream `Usage.cost.total`), per-model `modelUsage` (input/output/cache tokens), `num_turns` (counted from `turn_end` events), and `permission_denials` (populated from path-gated `beforeToolCall` blocks). The CLI's "Thought for Xs · N turns" footer reflects real numbers across both families instead of zeros for OpenAI.
• Thinking blocks survive translation. Reasoning content now becomes a proper `thinking` content block on the assistant SDKMessage (was folded into text on the OpenAI side). The desktop UI's "Claude is thinking" affordance keeps working; the empty-turn fallback still skips silently when there's no text, no tool call, and no thinking either.
• `beforeToolCall` for permission gating. Path-gated tools (Read/Write/Edit/Bash/NotebookEdit) flow through `security.promptForApproval` via pi-agent-core's `beforeToolCall` hook. Denials become `{ block: true, reason }` on pi's side and surface as `permission_denials` in the result message. `AskUserQuestion` is registered as a regular tool now (was an SDK preset interception path), so the question UX comes for free through the same channel.
• Sidecar persistence. The cross-fork transcript file moves from `.openai-state.json` → `.pi-state.json`. The loader still reads the legacy suffix as a one-time fallback so resumed sessions don't lose pi memory across the migration. Delete sweeps both.
• Anthropic env-resolution. `ANTHROPIC_AUTH_TOKEN` (wpcom proxy) and `ANTHROPIC_API_KEY` (direct API key) both populate the apiKey; `ANTHROPIC_BASE_URL` overrides the default `https://api.anthropic.com\`; `ANTHROPIC_CUSTOM_HEADERS` is parsed from the colon-delimited format `providers.ts` writes and merged into `Model.headers` so the wpcom `X-WPCOM-AI-Feature` header lands at the provider.
• Tests. `runtimes/openai/tests/` → `runtimes/pi/tests/` (with persistence test extended for the new + legacy suffix matrix). `tests/openai-runtime.test.ts` → `tests/pi-runtime.test.ts` with a new case verifying Claude dispatch builds an `anthropic-messages` Model. `tests/agent.test.ts` rewritten to assert each family routes to the right pi-ai `Model.api` rather than mocking the SDK `query()`. 1401 tests pass (up from 1396 before this branch's additions).

Testing Instructions

CLI smoke (after `npm run cli:build && node apps/cli/dist/cli/main.mjs ai`):

• Pick `gpt-5.5` from `/model` and run a turn — assistant text should render, the footer should show non-zero cost.
• Pick `claude-sonnet-4-6` from `/model` and run a turn — same checks. With Opus 4.7, watch for thinking blocks rendering as their own UI segment, not folded into text.
• Mid-session, swap families with `/model` (gpt-5.5 ↔ claude-sonnet-4-6). The conversation should continue (the new Agent rebuilds with the existing transcript), and the previous turn's tool results should still be visible to the model.
• Open the desktop UI's family-switch confirm dialog (apps/ui composer) — confirm the swap, run a turn, repeat. The cross-family resume guard is gone; both families share session ids now.

Local site interactions:

• Ask the agent to Read/Write a file outside the trusted root. Decline once → `permission_denials` shows up in the synthetic result. Allow always → the second access in the same directory doesn't re-prompt.
• Ask the agent to use `AskUserQuestion` (e.g. "ask me a question with three options"). The question picker should appear and the answer should land back in the assistant transcript.

Standalone MCP:

• Run `node apps/cli/dist/cli/main.mjs mcp` (non-TTY) and verify the stdio server starts and responds to a manual MCP `tools/list` request with the full Studio tool catalog.

Resume across forks (desktop):

• In the desktop app, start a new chat, run a turn, close + reopen — the `*.pi-state.json` sidecar should be present next to the JSONL and the agent should remember the prior turn.
• Resume an older session that has a `.openai-state.json` sidecar from before this branch — the legacy fallback should rehydrate it; the next turn writes a new `.pi-state.json`.

Pre-merge Checklist

• No TypeScript, React, or console errors (`npm run typecheck` clean).
• `npm test` passes (1401 tests).
• Cache-hit rate parity verified against the old Anthropic runtime — pi-ai's anthropic provider sets `cache_control: { type: 'ephemeral' }` on system + tools + last user content; check the wpcom proxy logs / `AssistantMessage.usage.cacheRead` are non-zero for follow-up turns.
• wpcom auth path verified end-to-end — the pi-ai anthropic provider has an OAuth token detection heuristic (looks for `sk-ant-oat`) that flips the auth header style; our wpcom access token doesn't match the heuristic, so the API-key path with our `X-WPCOM-AI-Feature` header should fire. Confirm by hitting the wpcom proxy and inspecting the request headers logged on the proxy side.
• Thinking-block UI rendering verified on a real Claude turn (Opus 4.7 with reasoning) and a real GPT 5.5 turn — both should produce visible "thinking" segments that don't overlap with the assistant text or get rendered as raw text.
• Skill appendix narrowing for Claude — `AUTO_APPLY_SKILLS` is currently `{ 'site-spec' }`, narrowed for the OpenAI side because GPT eagerly auto-runs tool-shaped skills. For Claude this is probably too restrictive; revisit in a follow-up tuning pass once we have signal from real Claude turns.
• `ALLOWED_TOOLS` still references `WebFetch`, `WebSearch`, `TodoRead`, `NotebookRead` from the SDK preset era — these are no-ops now (the unified runtime doesn't register them). Documented inline; tracked as a follow-up to either re-implement or drop from the list.

🤖 Generated with Claude Code

[chubes4]

I think this is a really great idea. It could eventually allow users the freedom to use any model they want.

[youknowriad]

Yes, but we need to ensure that there's no regressions. The pi based custom hardness has a different feeling that the agent SDK based one. I'm going to need a bit of testing to be able to land this.

[chubes4]

I tested this locally through the Studio eval runner with a Homeboy cross-rig bench comparing trunk/Claude Agent SDK against this branch/pi-agent-core.

The direction looks promising, but I found three pi adapter issues that made the initial result look faster while producing no usable assistant output on the WP.com Anthropic path:

1. The WP.com Anthropic proxy path needs Authorization: Bearer <token> in the pi model headers. Passing the token only as the Anthropic API key produced a pi assistant error: 401 Sorry, you are not allowed to do that.
2. pi assistant messages with stopReason: "error" need to be translated into an SDK result with subtype: "error_during_execution". Otherwise auth/provider failures look like successful empty runs.
3. Some successful pi responses surfaced assistant text on turn_end.message without a preceding message_end. The SDK compatibility layer should translate that message too, and dedupe when both events carry the same assistant message.

I patched those locally and added focused tests around auth headers, pi error messages, turn_end.message, and deduping. With those fixes, a no-tool identity prompt now works through both runtimes using claude-sonnet-4-6:

• Prompt: In one short sentence, tell me who you are. Do not call any tools.
• SDK runs: 7518, 7552, 8410 ms
• pi runs: 3310, 3000, 3384 ms
• Both paths: success_rate=1, turn_count=1, assistant_message_count=1, tool_call_count=0
• pi p50 was about 56% faster for this narrow no-tool case.

I would treat that as encouraging, not conclusive. Before landing this, I think we should also cover at least one tool-call scenario, one denial/error scenario, and one small site-building scenario, because the current proof only validates the simplest assistant-message path.

The local files I changed are apps/cli/ai/runtimes/pi/index.ts and apps/cli/ai/tests/pi-runtime.test.ts. Happy to turn that into a patch/PR if you want.

Edit: posted by Franklin, Chris's clanker agent, on Chris's behalf.

[youknowriad]

Closing this one, starting with less disruptive #3328