Move eval/ to scripts/eval/

[wojtekn]

Related issues

• Related to Add PromptFoo evaluation framework for Studio Code agent #3031

How AI was used in this PR

Claude Code performed the move, updated the path references, and verified the suite still runs (npm run eval -- -n 1 passes against the moved config).

Proposed Changes

The PromptFoo eval suite added in #3031 lives in a top-level eval/ directory. It's a dev tool, not an app or product code, so it doesn't warrant a project-root directory — this moves it under scripts/ alongside the other development tooling, keeping the repo root clean. No behavior change: npm run eval and npm run eval:view work exactly as before.

Testing Instructions

1. studio auth login
2. npm run eval -- -n 1
3. Verify the identity test passes (proves the moved config resolves the runner and grader paths).

Pre-merge Checklist

• Have you checked for TypeScript, React or other console errors?

🤖 Generated with Claude Code

[wpmobilebot]

📊 Performance Test Results

Comparing e2263ba vs trunk

app-size

Metric	trunk	e2263ba	Diff	Change
App Size (Mac)	1358.09 MB	1358.09 MB	+0.00 MB	⚪ 0.0%

site-editor

Metric	trunk	e2263ba	Diff	Change
load	1641 ms	1623 ms	18 ms	⚪ 0.0%

site-startup

Metric	trunk	e2263ba	Diff	Change
siteCreation	9029 ms	9048 ms	+19 ms	⚪ 0.0%
siteStartup	4412 ms	4410 ms	2 ms	⚪ 0.0%

---

Results are median values from multiple test runs.

Legend: 🟢 Improvement (faster) | 🔴 Regression (slower) | ⚪ No change (<50ms diff)

[epeicher]

Thanks @wojtekn for tidying up the repo. I have tested the evals, and they are executed without any issues. The warnings on the screenshot also happen on trunk. Changes LGTM!

[epeicher]

As a follow-up, this is Claude's explanation of the warnings that appear when running promptfoo. The gist of it is that they are harmless and actually a good thing, as it avoids caching previous unwanted results:

Could not find any valid files in the command: node ././apps/cli/dist/cli/eval-runner.mjs 
Could not hash any files for command node ../../apps/cli/dist/cli/eval-runner.mjs, caching will not be used

Claude Code explanation:

Yes — those warnings come from a quirk in promptfoo's exec: provider caching, and they're harmless. Here's exactly what happens (verified against the promptfoo 0.121.4 source that npx cached on your machine):

The mechanism. Before running your provider command, promptfoo tries to build a cache key by hashing any file it can find among the command's tokens (node, ../apps/cli/dist/cli/eval-runner.mjs). The catch is in src/providers/scriptCompletion.ts:

• The hashing step checks fs.existsSync(part) with plain relative paths, so it resolves them against the process working directory — the repo root, since npm run eval runs there.
• The execution step runs the command with cwd set to the config file's directory (eval/), which is why the relative path in eval/promptfoo.config.yaml is written relative to eval/.

So when hashing, promptfoo looks for /../apps/cli/dist/cli/eval-runner.mjs — outside the repo, doesn't exist — and node isn't a file in the cwd either. Zero hashable files produces the first warning ("Could not find any valid files"), and since the cache key is content-based, it then disables caching and prints the second one. The script itself still executes fine afterward because execution uses the eval/ cwd. It's an inconsistency in promptfoo (hash-cwd ≠ exec-cwd), not a problem with your setup.

Impact: effectively none — and arguably a good thing here. The only consequence is that promptfoo won't cache the runner's output between runs. For this eval suite you actually want that: the cases spin up the real agent and create/delete real sites, so a cached stdout from a previous identical run would skip the agent entirely and make the assertions meaningless.