Add daily eval CI workflow with Slack notifications by lezama · Pull Request #3042 · Automattic/studio

lezama · 2026-04-09T22:54:35Z

Summary

Adds a daily agent evaluation step to the existing Studio Buildkite pipeline. Runs automatically only on scheduled builds, otherwise it's behind a manual opt-in button.

Stacked on #3031 — merge that first.

Changes

.buildkite/pipeline.yml — Eval step gated on build.source == 'schedule' or EVAL_PIPELINE=true
.buildkite/commands/run-eval.sh — Installs deps, builds CLI, runs eval, posts results to Slack
apps/cli/ai/eval-runner.ts — Skip provider resolution when ANTHROPIC_API_KEY is already set (needed in CI where there's no Studio auth)

Setup (after merge)

An admin adds a schedule in the Studio pipeline settings:

Cron: 0 6 * * *
Branch: trunk
Env vars: EVAL_SLACK_CHANNEL=C08LZ79MKG9

ANTHROPIC_API_KEY and SLACK_TOKEN are already available from the pipeline's existing secrets. To trigger manually: new build with env var EVAL_PIPELINE=true.

Slack notification

✅ Agent eval: 3/3 passed
• agent identifies as WordPress Studio AI: ✅
• agent calls site_create and it succeeds: ✅
• agent asks permission before writing outside ~/Studio: ✅
View build

wojtekn · 2026-04-10T07:35:29Z

@lezama let's use Buildkite for any actions that build the app, and use GH actions only for basic stuff like static analysis, PR gardening, auto-merges, etc.

lezama · 2026-04-10T13:25:01Z

@wojtekn updated to use buildkite, but needs lots of review, not sure what I am doing.

… longer timeout

gcsecsey

Hey @lezama 👋 I'm checking in on this one since it has been idle for a bit. I found STU-1417, which has the broader goal of having an eval framework so we can track Studio AI quality over time, so this approach makes sense to me.

I tested this on Buildkite (build). I created a new build on this branch, using these env vars:

EVAL_PIPELINE=true
EVAL_TEST_FILTER=1

EVAL_PIPELINE=true makes the new Buildkite step to run, and EVAL_TEST_FILTER=1 runs only one eval case, to make the test quicker and cheaper. I also omitted setting EVAL_SLACK_CHANNEL to avoid sending a notification for now.

The new Agent Evaluation step was picked up correctly, dependencies installed, and the CLI built, and eval/results.json was uploaded.

However, the eval itself did not actually run successfully. PromptFoo reported provider errors for all test cases: Could not find any valid files in the command: node ../apps/cli/dist/cli/eval-runner.mjs

So it seems that the Buildkite wiring mostly works, but the eval command is being run from the wrong directory.

# Conflicts: # .buildkite/pipeline.yml # apps/cli/ai/eval-runner.ts

wpmobilebot · 2026-06-03T02:00:54Z

📊 Performance Test Results

Comparing 2b787f1 vs trunk

app-size

Metric	trunk	`2b787f1`	Diff	Change
App Size (Mac)	1315.41 MB	2319.47 MB	+1004.06 MB	🔴 76.3%

site-editor

Metric	trunk	`2b787f1`	Diff	Change
load	1077 ms	1071 ms	6 ms	⚪ 0.0%

site-startup

Metric	trunk	`2b787f1`	Diff	Change
siteCreation	6506 ms	6491 ms	15 ms	⚪ 0.0%
siteStartup	6488 ms	7005 ms	+517 ms	🔴 8.0%

Results are median values from multiple test runs.

Legend: 🟢 Improvement (faster) | 🔴 Regression (slower) | ⚪ No change (<50ms diff)

lezama · 2026-06-03T10:34:27Z

hey @gcsecsey , thanks for the reminder about this one!

I gave it a pass and rebase to bring it up to date, also pushed Add optional manual trigger to run the agent eval on a PR

In my local testing we need #3682 for tests to pass.

…-workflow

gcsecsey · 2026-06-18T16:04:12Z

Thanks @lezama for updating this, I merged in the trunk changes and also noticed that we're using the wrong paths, so I also pushed a fix for that.

The newly added Agent Evaluation step now runs, but fails on each test case to call the provider. I think we might be missing an ANTHROPIC_API_KEY env var that we check for in eval-runner.ts. Do you know maybe how we can add it to Buildkite?

Copilot

Pull request overview

Adds a scheduled (and optionally manual) Buildkite step to run the Studio Code agent PromptFoo eval suite and notify a Slack channel with pass/fail results, plus a small CI-oriented tweak to eval-runner environment resolution.

Changes:

Adds an “Agent Evaluation” step to .buildkite/pipeline.yml, intended to run on scheduled builds (and optionally on PRs via an input gate).
Introduces .buildkite/commands/run-eval.sh to install deps, build the CLI, run PromptFoo evals, and post a Slack summary.
Updates apps/cli/ai/eval-runner.ts to skip provider/env resolution when ANTHROPIC_API_KEY is already present.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 5 comments.

File	Description
`apps/cli/ai/eval-runner.ts`	Skips provider/env resolution when credentials are already present (CI path).
`.buildkite/pipeline.yml`	Adds an input gate + eval step wiring for scheduled or opt-in runs.
`.buildkite/commands/run-eval.sh`	New CI script to run PromptFoo evals and send a Slack notification from results JSON.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

iangmaia · 2026-06-18T17:38:21Z

+# Slack-standard good (#36a64f) and danger (#e01e5a) attachment colors.
+payload=$(jq --arg url "$RUN_URL" '
+  .results.stats as $s |
+  [.results.results[] |
+    "• " + (.testCase.description // .vars.caseId // "unknown") + ": " +
+    (if .success then "✅" else "❌" end)
+  ] | join("\n") as $lines |
+  (($s.successes + $s.failures + $s.errors) | tostring) as $total |
+  (if $s.failures == 0 and $s.errors == 0
+    then ":white_check_mark: Agent eval: " + ($s.successes | tostring) + "/" + $total + " passed"
+    else ":x: Agent eval: " + ($s.failures | tostring) + "/" + $total + " failed"
+  end) as $header |
+  (if $s.failures == 0 and $s.errors == 0 then "#36a64f" else "#e01e5a" end) as $color |
+  {
+    channel: $ENV.EVAL_SLACK_CHANNEL,
+    username: "Studio Eval",
+    icon_emoji: ":test_tube:",
+    attachments: [{
+      color: $color,
+      blocks: [
+        { type: "section", text: { type: "mrkdwn", text: $header } },
+        { type: "section", text: { type: "mrkdwn", text: $lines } },
+        { type: "context", elements: [{ type: "mrkdwn", text: ("<" + $url + "|View build>") }] }
+      ]
+    }]
+  }
+' "$RESULTS_FILE")
+
+if curl -sf -o /dev/null -X POST https://slack.com/api/chat.postMessage \
+    -H "Authorization: Bearer $SLACK_TOKEN" \
+    -H 'Content-Type: application/json; charset=utf-8' \
+    --data-binary "$payload"; then
+  echo "Sent to $EVAL_SLACK_CHANNEL"
+else
+  echo "Warning: Slack notification failed"
+fi


Just a thought, not sure if it would be a worthwhile improvement: I wonder if this couldn't be replaced by a Fastlane action call (since we have utils to send slack messages). At the same time, that would require a install_gems and using the Ruby runtime, which this script saves so 🤷

I think it's better to keep the curl version for now, to avoid pulling in install_gems + the Ruby runtime just for sending the Slack message.

to avoid pulling in install_gems + the Ruby runtime just for sending the Slack message.

I'll push back on this. Given the script runs on the mac queue, which already has Ruby available, I think the advantage of delegating the Slack plumbing to fastlane would be worth while: less custom code for us to maintain.

This sounds like a refactor, though, no need to block the PR for it.

…PI_KEY

gcsecsey · 2026-06-25T16:41:13Z

Hey @iangmaia 👋 thanks for the review! I think we have a blocker with the env vars, and I'm hoping you can help.

When I tested this on Buildkite, the step builds and runs fine, but every eval case fails on the provider call. My guess is that ANTHROPIC_API_KEY isn't in the environment.

Is there a mechanism we can use to expose ANTHROPIC_API_KEY and SLACK_TOKEN to the job as an env var? If not, could you add these as Buildkite secrets?

mokagio · 2026-06-26T05:26:41Z

+  exit "$eval_status"
+fi
+
+RUN_URL="${BUILDKITE_BUILD_URL:-https://buildkite.com}"


Might it be cleaner to fail if BUILDKITE_BUILD_URL is not available? It's a core expectation of the CI environment after all.

mokagio · 2026-06-26T05:34:49Z

+  # Agent evaluation. Runs automatically on the daily schedule (or with
+  # EVAL_PIPELINE=true), and optionally on a PR via the gate above.
+  # Setup details in scripts/eval/README.md.
+  - label: ":test_tube: Agent Evaluation"
+    key: agent_eval
+    depends_on: input-eval
+    agents:
+      queue: mac
+    command: bash .buildkite/commands/run-eval.sh
+    plugins: [$CI_TOOLKIT_PLUGIN, $NVM_PLUGIN]
+    artifact_paths:
+      - scripts/eval/results.json
+    timeout_in_minutes: 25
+    if: build.source == 'schedule' || build.env('EVAL_PIPELINE') == 'true' || (build.branch != 'trunk' && build.tag !~ /^v[0-9]+/)


If the intention is to run this step on a schedule, then with this setup where the only call is in pipeline.yml, the schedule would require running the whole pipeline "only" to get the eval result.

Is this acceptable?

The alternative is to have a dedicated pipeline, something like agent_eval_pipeline.yml with a step like this one and no if conditions, and configure the schedule to run that pipeline.

The downside would be that we'd need to duplicate the step between agent_eval_pipeline.yml and pipeline.yml (with the pipeline.yml version having all of the if checks it currently has minus build.source == 'schedule' and possibly build.env('EVAL_PIPELINE') == 'true'.

The downside would be that we'd need to duplicate the step between agent_eval_pipeline.yml and pipeline.yml (with the pipeline.yml version having all of the if checks it currently has minus build.source == 'schedule' and possibly build.env('EVAL_PIPELINE') == 'true'.

Thinking about it, but I'm not 100% sure, given the depends_on: input-eval, we might do entirely away with the if in this step, and rely entirely on the conditions defined in the input-eval step.

If the intention is to run this step on a schedule, then with this setup where the only call is in pipeline.yml, the schedule would require running the whole pipeline "only" to get the eval result.

I'm not sure about the motivation for this change, but I'm not sure it's imperative that we run this on a schedule, but to run this often enough to catch any regressions. @lezama please correct me if I'm wrong.

Would it make sense to run this step automatically for builds from trunk? That'd ensure it runs at least daily.

mokagio · 2026-06-26T05:36:47Z

Is there a mechanism we can use to expose ANTHROPIC_API_KEY and SLACK_TOKEN to the job as an env var? If not, could you add these as Buildkite secrets?

@gcsecsey yes there is. Let's sync out of GitHub about how.

(Update: Internal ref p1782452433458179-slack-C06DRMD6VPZ)

wojtekn assigned lezama Apr 10, 2026

lezama force-pushed the add-eval-ci-workflow branch 2 times, most recently from 673db65 to 42695c8 Compare April 10, 2026 13:01

lezama force-pushed the add-promptfoo-evaluation branch from bc6cc2a to e15062d Compare April 10, 2026 13:02

lezama force-pushed the add-eval-ci-workflow branch from 42695c8 to fda473f Compare April 10, 2026 13:02

lezama force-pushed the add-promptfoo-evaluation branch from e15062d to 4c01b4c Compare April 15, 2026 13:06

Base automatically changed from add-promptfoo-evaluation to trunk April 21, 2026 18:07

lezama added 2 commits April 23, 2026 21:06

Support ANTHROPIC_API_KEY from env (skip Studio auth in CI)

6e0cedc

Add daily eval Buildkite step with Slack notifier

f6aa7d9

lezama force-pushed the add-eval-ci-workflow branch from fda473f to f6aa7d9 Compare April 24, 2026 01:07

Polish eval CI: bash array, repo-relative paths, tighter Slack guard,…

bcba94b

… longer timeout

lezama marked this pull request as ready for review April 24, 2026 14:53

gcsecsey reviewed Jun 2, 2026

View reviewed changes

Merge remote-tracking branch 'origin/trunk' into add-eval-ci-workflow

e317d8b

# Conflicts: # .buildkite/pipeline.yml # apps/cli/ai/eval-runner.ts

lezama added 2 commits June 3, 2026 07:21

Add optional manual trigger to run the agent eval on a PR

3415f92

Merge branch 'trunk' into add-eval-ci-workflow

7891ca8

lezama requested review from draganescu and youknowriad June 3, 2026 10:34

Merge branch 'trunk' of github.com:Automattic/studio into add-eval-ci…

af356bb

…-workflow

gcsecsey requested a review from a team as a code owner June 18, 2026 14:54

Fix eval CI paths after eval/ moved to scripts/eval/ (#3775)

1b05a52

iangmaia requested a review from Copilot June 18, 2026 16:58

Copilot started reviewing on behalf of iangmaia June 18, 2026 16:58 View session

Copilot AI reviewed Jun 18, 2026

View reviewed changes

Comment thread .buildkite/pipeline.yml

Comment thread .buildkite/commands/run-eval.sh

Comment thread .buildkite/commands/run-eval.sh Outdated

Comment thread apps/cli/ai/eval-runner.ts Outdated

Comment thread .buildkite/commands/run-eval.sh

iangmaia reviewed Jun 18, 2026

View reviewed changes

gcsecsey added 4 commits June 25, 2026 17:11

Send eval Slack notification even when assertions fail

7bd31c3

Count eval errors as failures in Slack summary header

8592071

Verify Slack API response ok field instead of trusting HTTP status

244f1bd

Only bypass eval provider resolution in CI, not for local ANTHROPIC_A…

2b787f1

…PI_KEY

mokagio reviewed Jun 26, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Add daily eval CI workflow with Slack notifications - #3042

Add daily eval CI workflow with Slack notifications#3042
lezama wants to merge 12 commits into
trunkfrom
add-eval-ci-workflow

lezama commented Apr 9, 2026 •

edited by gcsecsey

Loading

wojtekn commented Apr 10, 2026

lezama commented Apr 10, 2026

gcsecsey left a comment •

edited

Loading

wpmobilebot commented Jun 3, 2026 •

edited

Loading

lezama commented Jun 3, 2026

gcsecsey commented Jun 18, 2026

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

iangmaia Jun 18, 2026

gcsecsey Jun 25, 2026

mokagio Jun 26, 2026

gcsecsey commented Jun 25, 2026

mokagio Jun 26, 2026

mokagio Jun 26, 2026

mokagio Jun 26, 2026

gcsecsey Jun 29, 2026

mokagio commented Jun 26, 2026 •

edited

Loading

Labels

7 participants

Uh oh!

Conversation

lezama commented Apr 9, 2026 • edited by gcsecsey Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Changes

Setup (after merge)

Slack notification

wojtekn commented Apr 10, 2026

lezama commented Apr 10, 2026

gcsecsey left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

wpmobilebot commented Jun 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

📊 Performance Test Results

app-size

site-editor

site-startup

lezama commented Jun 3, 2026

gcsecsey commented Jun 18, 2026

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

iangmaia Jun 18, 2026

Choose a reason for hiding this comment

gcsecsey Jun 25, 2026

Choose a reason for hiding this comment

mokagio Jun 26, 2026

Choose a reason for hiding this comment

gcsecsey commented Jun 25, 2026

mokagio Jun 26, 2026

Choose a reason for hiding this comment

mokagio Jun 26, 2026

Choose a reason for hiding this comment

mokagio Jun 26, 2026

Choose a reason for hiding this comment

gcsecsey Jun 29, 2026

Choose a reason for hiding this comment

mokagio commented Jun 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Labels

7 participants

lezama commented Apr 9, 2026 •

edited by gcsecsey

Loading

gcsecsey left a comment •

edited

Loading

wpmobilebot commented Jun 3, 2026 •

edited

Loading

mokagio commented Jun 26, 2026 •

edited

Loading