Skip to content

Fix screenshot eval assertion to match the tool's JPEG output - #3682

Merged
lezama merged 1 commit into
trunkfrom
fix-screenshot-eval-jpeg-assertion
Jun 3, 2026
Merged

Fix screenshot eval assertion to match the tool's JPEG output#3682
lezama merged 1 commit into
trunkfrom
fix-screenshot-eval-jpeg-assertion

Conversation

@lezama

@lezama lezama commented Jun 3, 2026

Copy link
Copy Markdown
Contributor

Related issues

None.

How AI was used in this PR

The code change and this PR description were drafted with Claude Code; the diagnosis was confirmed by running the eval suite locally.

Proposed Changes

The screenshot-all-timing agent eval has been failing because its assertion still expected PNG screenshots, while take_screenshot now produces JPEG (screenshot-desktop.jpg / screenshot-mobile.jpg, image/jpeg). The agent behavior was correct all along — one fast viewport: "all" capture of desktop and mobile — but the check rejected the payload on the file name/format and parsed dimensions as PNG.

This updates the assertion to validate the real invariant rather than the encoding:

  • Keys the expected width by viewport (desktop → 1040, mobile → 390) instead of the file extension, so it no longer breaks if the format changes again.
  • Reads pixel dimensions from either a PNG or a JPEG buffer.

The eval suite is green again, so the daily eval reports an accurate signal instead of a standing false failure.

Testing Instructions

  1. From eval/, run the screenshot case:
    npx promptfoo@0.121.4 eval -c promptfoo.config.yaml --no-cache --filter-pattern "screenshot verification"
  2. Confirm it passes (it captures one viewport: "all" screenshot and validates the desktop/mobile JPEG dimensions).

Pre-merge Checklist

  • Have you checked for similar PRs that may already address this?
  • Verified locally (the case passes).
  • Reviewed by a teammate.
@wpmobilebot

Copy link
Copy Markdown
Collaborator

📊 Performance Test Results

Comparing a907ccf vs trunk

app-size

Metric trunk a907ccf Diff Change
App Size (Mac) 1334.62 MB 1330.69 MB 3.93 MB 🟢 -0.3%

site-editor

Metric trunk a907ccf Diff Change
load 1633 ms 1634 ms +1 ms ⚪ 0.0%

site-startup

Metric trunk a907ccf Diff Change
siteCreation 9053 ms 9032 ms 21 ms ⚪ 0.0%
siteStartup 4427 ms 4420 ms 7 ms ⚪ 0.0%

Results are median values from multiple test runs.

Legend: 🟢 Improvement (faster) | 🔴 Regression (slower) | ⚪ No change (<50ms diff)

@lezama
lezama merged commit d2473ad into trunk Jun 3, 2026
12 checks passed
@lezama
lezama deleted the fix-screenshot-eval-jpeg-assertion branch June 3, 2026 12:14

@sejas sejas left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for updating the eval 🙌

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

4 participants