Skip to content

Reliable visual verification: sharp screenshots + measure_elements + anti-spiral guidance - #3647

Draft
lezama wants to merge 4 commits into
trunkfrom
improve-agent-visual-verification
Draft

Reliable visual verification: sharp screenshots + measure_elements + anti-spiral guidance#3647
lezama wants to merge 4 commits into
trunkfrom
improve-agent-visual-verification

Conversation

@lezama

@lezama lezama commented May 29, 2026

Copy link
Copy Markdown
Contributor

Related issues

  • Related to #

How AI was used in this PR

Code, the new tool, the skill guidance, and this description were drafted with Claude Code and reviewed by me. The motivation came from watching a real studio code build session spiral for ~24 minutes (164 turns) on a layout bug that did not exist; the changes here target that failure mode directly. Browser logic and the formatter were smoke-tested live.

Proposed Changes

In a real build session, the agent spent ~24 minutes "fixing" a grid that was already correct. Root cause: take_screenshot is always full-page, so a ~6000px-tall page becomes a ~273px-wide thumbnail after the vision pipeline downscales it — three columns blur into two, cards look "missing," and the agent chases phantom CSS bugs, eventually degrading correct CSS (repeat(3, 1fr)auto-fit minmax(120px)) to make a blurry screenshot look different.

This gives the agent reliable ways to see ground truth, and tells it when not to trust its eyes:

  • Sharp viewport screenshotstake_screenshot gains fullPage: false: a crisp viewport-height slice (paged with offset) that isn't downscaled, so fine layout detail stays legible.
  • measure_elements — a new tool that returns how elements actually render (box + computed grid-template-columns, flex-wrap, display, gaps, width, overflow) for a CSS selector. "Is it really 3 columns?" becomes a fact, not a guess.
  • visual-design guidance — don't count columns/cards from a full-page shot; verify with a viewport shot or measure_elements; stop after two failed fixes of the same symptom; and never degrade real-browser-correct CSS to satisfy a screenshot.

Why a Proof of Concept: the screenshot change is low-risk (full-page stays the default), but measure_elements is a new tool surface and the prompt/skill guidance is worth a team read for tone and placement. Open question: whether to make fullPage: false the default for in-build verification rather than opt-in.

Testing Instructions

  1. npm run cli:build
  2. Build a long page, then verify a grid section with take_screenshot fullPage: false + offset — the slice should be crisp enough to count columns. Compare to the full-page shot.
  3. measure_elements on the running site with a selector like .wp-block-columns — confirm it returns the rendered grid-template-columns.
  4. npm test -- apps/cli/ai/tests/measure-elements.test.ts — unit tests for the result formatter (match / no-match / truncation).

Pre-merge Checklist

  • Have you checked for TypeScript, React or other console errors?
Adds a sharp viewport-height screenshot mode (take_screenshot fullPage:false),
a measure_elements tool that reports rendered box + layout computed styles for
a selector, and visual-design guidance to stop layout-debugging spirals:
don't judge fine layout from downscaled full-page shots, verify ground truth,
stop after two failed fixes, and never degrade correct CSS to satisfy a
screenshot.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
lezama and others added 3 commits June 3, 2026 08:20
Phrase the layout-verification rules at the capability level (viewport-height
capture, computed-style measurement) with a thin 'In Studio' overlay naming
the actual tools. Keeps the portable knowledge liftable into a shared,
cross-harness skill while Studio still maps it to take_screenshot/measure_elements.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

1 participant