Add PromptFoo evaluation framework for Studio Code agent

[lezama]

Summary

Adds a PromptFoo evaluation framework for the Studio Code AI agent. A custom eval runner hooks into startAiAgent() to test the full agent loop — tool calls, permission flows, and output quality.

How it works

npm run eval → promptfoo → eval-runner.ts → startAiAgent() → structured JSON → assertions

The runner returns raw data (tool calls, tool results, text segments, permission questions). All assertion logic stays in the YAML config.

A custom grader provider calls Claude Haiku via the WP.com AI proxy using Studio's existing auth token — no separate API key needed for local development.

Test cases

Test	What it checks	Type
identity	Agent identifies as WordPress Studio AI	LLM judge
site-creation	site_create tool called and succeeds	deterministic
security	Agent asks permission before writing outside ~/Studio	deterministic

Changes

• apps/cli/ai/eval-runner.ts — Runner that hooks into startAiAgent(), captures tool calls/results/text/questions, returns JSON
• apps/cli/vite.config.dev.ts — eval-runner as dev-only Vite entry point (not in production builds)
• eval/grader-provider.mjs — PromptFoo grader that calls Claude Haiku via WP.com proxy for LLM judge assertions
• eval/promptfoo.config.yaml — 3 test cases with deterministic + LLM judge assertions
• eval/README.md — Usage docs
• package.json — npm run eval and npm run eval:view scripts

Usage

studio auth login
npm run eval               # builds CLI, runs all 3 tests (~30s)
npm run eval -- -n 1       # run first test only
npm run eval:view          # view results in browser

Follow-up

Daily CI workflow with Slack notifications — shipping separately once testable on trunk.

Testing Instructions

1. studio auth login
2. npm run eval
3. Verify 3/3 tests pass
4. npm run eval:view to inspect results

Quality gates

• ESLint: pass
• TypeScript: pass
• Tests: 70/70 pass
• Eval suite: 3/3 pass (32s)

[wpmobilebot]

📊 Performance Test Results

Comparing 3bc7584 vs trunk

app-size

Metric	trunk	3bc7584	Diff	Change
App Size (Mac)	1490.96 MB	1490.95 MB	0.01 MB	⚪ 0.0%

site-editor

Metric	trunk	3bc7584	Diff	Change
load	1934 ms	1932 ms	2 ms	⚪ 0.0%

site-startup

Metric	trunk	3bc7584	Diff	Change
siteCreation	10129 ms	9128 ms	1001 ms	🟢 -9.9%
siteStartup	5959 ms	4946 ms	1013 ms	🟢 -17.0%

---

Results are median values from multiple test runs.

Legend: 🟢 Improvement (faster) | 🔴 Regression (slower) | ⚪ No change (<50ms diff)

[sejas]

Related issue STU-1417

[sejas]

Thanks for adding Promptfoo and first tests. I executed npm run eval a few times and it worked as expected.

I left a suggestion about using Promptfoo as a installed devDependency vs npx .

[sejas]

Should we install a specific version as a devDependency? I’m not sure whether using the npx command directly is best practice.

[youknowriad]

I don't mind npx to lazy-install it on CI but at least we should fix the version I think.

[youknowriad]

anything blocking the merge of this PR, I'd be good to start working and adding some of these tests. and probably send the daily report on the studio dev channel.