Skip to content

feat(studio): consume eval-runner failure-capture fields - #57

Merged
chubes4 merged 1 commit into
mainfrom
bench-consume-eval-runner-failure-fields
May 4, 2026
Merged

feat(studio): consume eval-runner failure-capture fields#57
chubes4 merged 1 commit into
mainfrom
bench-consume-eval-runner-failure-fields

Conversation

@chubes4

@chubes4 chubes4 commented May 4, 2026

Copy link
Copy Markdown
Owner

Summary

Pairs with Automattic/studio#3330, which adds two new fields to the eval-runner result JSON:

  • result.error: string | null — set when an exception is caught inside the message loop (auth blip, MCP crash, network error, SDK throw, etc.)
  • result.timedOut: boolean — set when the timeout callback fires before query.interrupt()

This PR teaches the Studio bench scripts to consume both fields.

Why

Today the bench gate is result.success === true && !result.error. On current trunk Studio, result.error is undefined outside the rare top-level catch path, so the AND degenerates to result.success === true. Timeouts are indistinguishable from clean model failures — both surface as success: false with no other differentiator.

After Studio#3330, the eval-runner emits both fields consistently, and the bench can:

  • Distinguish timeouts from agent failures. A timed-out 5-minute run and a model that gave up at turn 3 today look identical in metrics. With this PR, timed_out is a separate metric.
  • Distinguish runner-side exceptions from agent failures. A run where the model finished cleanly but the runner caught a recoverable exception today degrades to a stripped-down { success: false, error } JSON, losing all timings and tool history. After Studio#3330 the structured result is preserved with error set, and this PR's agent_runner_error metric makes it visible.
  • Show the actual failure mode in error messages. studio-agent-runtime and studio-agent-site-info now print timed out after Nms / exception: ... / exit=N instead of always falling back to exit=N.

Diff

Three bench files, ~30 lines:

  • studio-agent-runtime.bench.mjs and studio-agent-site-info.bench.mjs: failure-detail message branches on result.timedOut and result.error. Lifts the timeout literal into a named constant so the message stays in sync with the actual budget.
  • studio-agent-site-build.bench.mjs: agentSucceeded gate now requires !result.timedOut. Adds timed_out and agent_runner_error metrics so timeout regressions are visible separately from agent failures.

Backwards compatibility

On Studio versions older than #3330, both result.error and result.timedOut are nullish (or absent). The new gates degrade safely: agentTimedOut resolves to false, !result.error resolves to true, and the bench behaves exactly as it did before. New metrics emit 0 instead of misleading values.

This means the rig PR can land independently of Studio#3330 — it's strictly additive on old Studio, fully active on new Studio.

Refs

AI assistance

  • AI assistance: Yes
  • Tool(s): Claude Code (Sonnet 4.5)
  • Used for: Drafted the gate-tightening, named-constant refactor, and failure-detail messages under Chris's direction.
Pairs with Automattic/studio#3330, which adds two new fields to the
eval-runner result JSON:

- result.error: string | null  — set when an exception is caught
  inside the message loop (auth blip, MCP crash, network error, SDK
  throw, etc.)
- result.timedOut: boolean     — set when the timeout callback fires
  before query.interrupt()

Bench changes:

- studio-agent-runtime + studio-agent-site-info: failure-detail
  message now distinguishes 'timed out after Nms', 'exception: ...',
  and 'exit=N'. Lifts the timeout literal into a named constant so
  the message stays in sync with the actual budget.
- studio-agent-site-build: agentSucceeded gate now also requires
  !result.timedOut. Adds two new metrics:
  - timed_out (1 when the run hit the time budget)
  - agent_runner_error (1 when an exception landed in result.error)
  These categorize regressions that today look like generic agent
  failures.

Backwards-compatible: on Studio versions older than #3330, both
fields are nullish and the bench behaves exactly as it did before.

Refs: Automattic/studio#3330, Automattic/studio#3262, Automattic/studio#3273

## AI assistance
- **AI assistance:** Yes
- **Tool(s):** Claude Code (Sonnet 4.5)
- **Used for:** Drafted the gate-tightening, the constants, and the
  failure-detail messages under Chris's direction.
@chubes4
chubes4 merged commit 1c8595a into main May 4, 2026
@chubes4
chubes4 deleted the bench-consume-eval-runner-failure-fields branch July 4, 2026 18:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

1 participant