feat(studio): consume eval-runner failure-capture fields - #57
Merged
Conversation
Pairs with Automattic/studio#3330, which adds two new fields to the eval-runner result JSON: - result.error: string | null — set when an exception is caught inside the message loop (auth blip, MCP crash, network error, SDK throw, etc.) - result.timedOut: boolean — set when the timeout callback fires before query.interrupt() Bench changes: - studio-agent-runtime + studio-agent-site-info: failure-detail message now distinguishes 'timed out after Nms', 'exception: ...', and 'exit=N'. Lifts the timeout literal into a named constant so the message stays in sync with the actual budget. - studio-agent-site-build: agentSucceeded gate now also requires !result.timedOut. Adds two new metrics: - timed_out (1 when the run hit the time budget) - agent_runner_error (1 when an exception landed in result.error) These categorize regressions that today look like generic agent failures. Backwards-compatible: on Studio versions older than #3330, both fields are nullish and the bench behaves exactly as it did before. Refs: Automattic/studio#3330, Automattic/studio#3262, Automattic/studio#3273 ## AI assistance - **AI assistance:** Yes - **Tool(s):** Claude Code (Sonnet 4.5) - **Used for:** Drafted the gate-tightening, the constants, and the failure-detail messages under Chris's direction.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Pairs with Automattic/studio#3330, which adds two new fields to the eval-runner result JSON:
result.error: string | null— set when an exception is caught inside the message loop (auth blip, MCP crash, network error, SDK throw, etc.)result.timedOut: boolean— set when the timeout callback fires beforequery.interrupt()This PR teaches the Studio bench scripts to consume both fields.
Why
Today the bench gate is
result.success === true && !result.error. On current trunk Studio,result.errorisundefinedoutside the rare top-level catch path, so the AND degenerates toresult.success === true. Timeouts are indistinguishable from clean model failures — both surface assuccess: falsewith no other differentiator.After Studio#3330, the eval-runner emits both fields consistently, and the bench can:
timed_outis a separate metric.{ success: false, error }JSON, losing all timings and tool history. After Studio#3330 the structured result is preserved witherrorset, and this PR'sagent_runner_errormetric makes it visible.studio-agent-runtimeandstudio-agent-site-infonow printtimed out after Nms/exception: .../exit=Ninstead of always falling back toexit=N.Diff
Three bench files, ~30 lines:
studio-agent-runtime.bench.mjsandstudio-agent-site-info.bench.mjs: failure-detail message branches onresult.timedOutandresult.error. Lifts the timeout literal into a named constant so the message stays in sync with the actual budget.studio-agent-site-build.bench.mjs:agentSucceededgate now requires!result.timedOut. Addstimed_outandagent_runner_errormetrics so timeout regressions are visible separately from agent failures.Backwards compatibility
On Studio versions older than #3330, both
result.errorandresult.timedOutare nullish (or absent). The new gates degrade safely:agentTimedOutresolves tofalse,!result.errorresolves totrue, and the bench behaves exactly as it did before. New metrics emit0instead of misleading values.This means the rig PR can land independently of Studio#3330 — it's strictly additive on old Studio, fully active on new Studio.
Refs
firstToolError,toolEvents,phaseTimingsMs)AI assistance