Add daily eval CI workflow with Slack notifications #3042

mokagio · 2026-06-26T05:26:41Z

Might it be cleaner to fail if BUILDKITE_BUILD_URL is not available? It's a core expectation of the CI environment after all.

iangmaia · 2026-06-18T17:38:21Z

Just a thought, not sure if it would be a worthwhile improvement: I wonder if this couldn't be replaced by a Fastlane action call (since we have utils to send slack messages). At the same time, that would require a install_gems and using the Ruby runtime, which this script saves so 🤷

I think it's better to keep the curl version for now, to avoid pulling in install_gems + the Ruby runtime just for sending the Slack message.

to avoid pulling in install_gems + the Ruby runtime just for sending the Slack message.

I'll push back on this. Given the script runs on the mac queue, which already has Ruby available, I think the advantage of delegating the Slack plumbing to fastlane would be worth while: less custom code for us to maintain.

This sounds like a refactor, though, no need to block the PR for it.

mokagio · 2026-06-26T05:34:49Z

If the intention is to run this step on a schedule, then with this setup where the only call is in pipeline.yml, the schedule would require running the whole pipeline "only" to get the eval result.

Is this acceptable?

The alternative is to have a dedicated pipeline, something like agent_eval_pipeline.yml with a step like this one and no if conditions, and configure the schedule to run that pipeline.

The downside would be that we'd need to duplicate the step between agent_eval_pipeline.yml and pipeline.yml (with the pipeline.yml version having all of the if checks it currently has minus build.source == 'schedule' and possibly build.env('EVAL_PIPELINE') == 'true'.

The downside would be that we'd need to duplicate the step between agent_eval_pipeline.yml and pipeline.yml (with the pipeline.yml version having all of the if checks it currently has minus build.source == 'schedule' and possibly build.env('EVAL_PIPELINE') == 'true'.

Thinking about it, but I'm not 100% sure, given the depends_on: input-eval, we might do entirely away with the if in this step, and rely entirely on the conditions defined in the input-eval step.

If the intention is to run this step on a schedule, then with this setup where the only call is in pipeline.yml, the schedule would require running the whole pipeline "only" to get the eval result.

I'm not sure about the motivation for this change, but I'm not sure it's imperative that we run this on a schedule, but to run this often enough to catch any regressions. @lezama please correct me if I'm wrong.

Would it make sense to run this step automatically for builds from trunk? That'd ensure it runs at least daily.

-Original file line number
+Diff line change
@@ -0,0 +1,78 @@
+    #!/usr/bin/env bash
+    set -euo pipefail
+    RESULTS_FILE="scripts/eval/results.json"
+    echo '--- :npm: Install Node dependencies'
+    bash .buildkite/commands/install-node-dependencies.sh
+    echo '--- :hammer: Build CLI'
+    npm run cli:build
+    echo '--- :test_tube: Run agent evaluation'
+    eval_args=( --no-cache --output "$RESULTS_FILE" )
+    if [[ -n "${EVAL_TEST_FILTER:-}" ]]; then
+      if ! [[ "$EVAL_TEST_FILTER" =~ ^[0-9]+$ ]]; then
+        echo "Error: EVAL_TEST_FILTER must be a number, got: $EVAL_TEST_FILTER"
+        exit 1
+      fi
+      eval_args+=( -n "$EVAL_TEST_FILTER" )
+    fi
+    # promptfoo exits non-zero when assertions fail. Capture the status instead of
+    # letting `set -e` abort here, so we still post the Slack notification (the
+    # failure case is exactly when we want it) and then exit with the eval's status.
+    eval_status=0
+    npx promptfoo@0.121.4 eval -c scripts/eval/promptfoo.config.yaml "${eval_args[@]}" || eval_status=$?
+    echo '--- :slack: Send Slack notification'
+    if [[ -z "${EVAL_SLACK_CHANNEL:-}" || -z "${SLACK_TOKEN:-}" || ! -f "$RESULTS_FILE" ]]; then
+      echo "Skipping Slack notification (missing channel/token/results)"
+      exit "$eval_status"
+    fi
+    RUN_URL="${BUILDKITE_BUILD_URL:-https://buildkite.com}"
+    # Slack-standard good (#36a64f) and danger (#e01e5a) attachment colors.
+    payload=$(jq --arg url "$RUN_URL" '
+      .results.stats as $s |
+      [.results.results[] |
+        "• " + (.testCase.description // .vars.caseId // "unknown") + ": " +
+        (if .success then "✅" else "❌" end)
+      ] | join("\n") as $lines |
+      (($s.successes + $s.failures + $s.errors) | tostring) as $total |
+      (if $s.failures == 0 and $s.errors == 0
+        then ":white_check_mark: Agent eval: " + ($s.successes | tostring) + "/" + $total + " passed"
+        else ":x: Agent eval: " + (($s.failures + $s.errors) | tostring) + "/" + $total + " failed"
+      end) as $header |
+      (if $s.failures == 0 and $s.errors == 0 then "#36a64f" else "#e01e5a" end) as $color |
+      {
+        channel: $ENV.EVAL_SLACK_CHANNEL,
+        username: "Studio Eval",
+        icon_emoji: ":test_tube:",
+        attachments: [{
+          color: $color,
+          blocks: [
+            { type: "section", text: { type: "mrkdwn", text: $header } },
+            { type: "section", text: { type: "mrkdwn", text: $lines } },
+            { type: "context", elements: [{ type: "mrkdwn", text: ("<" + $url + "|View build>") }] }
+          ]
+        }]
+      }
+    ' "$RESULTS_FILE")
+    # Don't use curl -f: chat.postMessage returns HTTP 200 with {"ok":false,"error":...}
+    # for app-level problems (bad channel, bad token), which -f would treat as success.
+    # Read the body and check .ok instead.
+    response=$(curl -s -X POST https://slack.com/api/chat.postMessage \
+      -H "Authorization: Bearer $SLACK_TOKEN" \
+      -H 'Content-Type: application/json; charset=utf-8' \
+      --data-binary "$payload" || true)
+    if jq -e '.ok == true' <<< "$response" >/dev/null 2>&1; then
+      echo "Sent to $EVAL_SLACK_CHANNEL"
+    else
+      reason=$(jq -r '.error // empty' <<< "$response" 2>/dev/null || true)
+      echo "Warning: Slack notification failed: ${reason:-no/invalid response}"
+    fi
+    # Preserve the eval outcome so a failing eval still fails the CI step.
+    exit "$eval_status"

-Original file line number
+Diff line change
@@ Expand Up / @@ -248,6 +248,30 @@ steps: @@
               - x64
               - arm64
+      # Optional manual trigger to run the agent eval on a PR. On scheduled builds
+      # (and EVAL_PIPELINE runs) this gate is skipped — a skipped dependency counts
+      # as passed — so the eval below runs automatically. On a PR it shows a button
+      # to opt in.
+      - input: ":test_tube: Run agent eval?"
+        prompt: "Run the Studio Code agent evaluation for this PR?"
+        key: input-eval
+        if: build.source != 'schedule' && build.env('EVAL_PIPELINE') != 'true' && build.branch != 'trunk' && build.tag !~ /^v[0-9]+/
+      # Agent evaluation. Runs automatically on the daily schedule (or with
+      # EVAL_PIPELINE=true), and optionally on a PR via the gate above.
+      # Setup details in scripts/eval/README.md.
+      - label: ":test_tube: Agent Evaluation"
+        key: agent_eval
+        depends_on: input-eval
+        agents:
+          queue: mac
+        command: bash .buildkite/commands/run-eval.sh
+        plugins: [$CI_TOOLKIT_PLUGIN, $NVM_PLUGIN]
+        artifact_paths:
+          - scripts/eval/results.json
+        timeout_in_minutes: 25
+        if: build.source == 'schedule' || build.env('EVAL_PIPELINE') == 'true' || (build.branch != 'trunk' && build.tag !~ /^v[0-9]+/)
       - input: "🚦 Build for Linux?"
         prompt: "Do you want to build Linux dev binaries for this PR?"
         key: input-dev-linux
@@ Expand Down @@

-Original file line number
+Diff line change
@@ Expand Up / @@ -134,21 +134,24 @@ async function runEval( input: EvalRunnerInput ) { @@
     	const phaseTimingsMs: Record< string, number > = {};
     	let phaseStartedAt = Date.now();
-    	let aiProvider: AiProviderId = await resolveInitialAiProvider();
-    	phaseTimingsMs.resolve_initial_provider_ms = Date.now() - phaseStartedAt;
-    	phaseStartedAt = Date.now();
-    	aiProvider = ( await resolveUnavailableAiProvider( aiProvider ) ) ?? aiProvider;
-    	phaseTimingsMs.resolve_unavailable_provider_ms = Date.now() - phaseStartedAt;
-    	phaseStartedAt = Date.now();
-    	const aiEnvironment = await resolveAiEnvironment( aiProvider );
-    	phaseTimingsMs.resolve_ai_environment_ms = Date.now() - phaseStartedAt;
-    	const env = {
-    		...( process.env as Record< string, string > ),
-    		...aiEnvironment,
-    	};
+    	// In CI, ANTHROPIC_API_KEY is injected and used directly. Locally we always
+    	// resolve via Studio's auth so `npm run eval` exercises the same provider path
+    	// (incl. the WP.com proxy) the app uses — even if the developer happens to have
+    	// ANTHROPIC_API_KEY set in their environment.
+    	const env = { ...( process.env as Record< string, string > ) };
+    	const useInjectedApiKey = !! env.ANTHROPIC_API_KEY && ( !! env.CI || !! env.BUILDKITE );
+    	if ( ! useInjectedApiKey ) {
+    		let aiProvider: AiProviderId = await resolveInitialAiProvider();
+    		phaseTimingsMs.resolve_initial_provider_ms = Date.now() - phaseStartedAt;
+    		phaseStartedAt = Date.now();
+    		aiProvider = ( await resolveUnavailableAiProvider( aiProvider ) ) ?? aiProvider;
+    		phaseTimingsMs.resolve_unavailable_provider_ms = Date.now() - phaseStartedAt;
+    		phaseStartedAt = Date.now();
+    		Object.assign( env, await resolveAiEnvironment( aiProvider ) );
+    		phaseTimingsMs.resolve_ai_environment_ms = Date.now() - phaseStartedAt;
+    	}
     	// Allow running inside a Claude Code session
     	delete env.CLAUDECODE;
@@ Expand Down @@

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Add daily eval CI workflow with Slack notifications #3042

Diff view

Diff view

There are no files selected for viewing

mokagio Jun 26, 2026

iangmaia Jun 18, 2026

gcsecsey Jun 25, 2026

mokagio Jun 26, 2026

Uh oh!

mokagio Jun 26, 2026

mokagio Jun 26, 2026

gcsecsey Jun 29, 2026

Uh oh!

Add daily eval CI workflow with Slack notifications #3042

Are you sure you want to change the base?

Add daily eval CI workflow with Slack notifications #3042

Uh oh!

Uh oh!

Diff view

Diff view

There are no files selected for viewing

mokagio Jun 26, 2026

Choose a reason for hiding this comment

iangmaia Jun 18, 2026

Choose a reason for hiding this comment

gcsecsey Jun 25, 2026

Choose a reason for hiding this comment

mokagio Jun 26, 2026

Choose a reason for hiding this comment

Uh oh!

mokagio Jun 26, 2026

Choose a reason for hiding this comment

mokagio Jun 26, 2026

Choose a reason for hiding this comment

gcsecsey Jun 29, 2026

Choose a reason for hiding this comment