Reliable visual verification: sharp screenshots + measure_elements + anti-spiral guidance - #3647
Draft
lezama wants to merge 4 commits into
Draft
Reliable visual verification: sharp screenshots + measure_elements + anti-spiral guidance#3647lezama wants to merge 4 commits into
lezama wants to merge 4 commits into
Conversation
Adds a sharp viewport-height screenshot mode (take_screenshot fullPage:false), a measure_elements tool that reports rendered box + layout computed styles for a selector, and visual-design guidance to stop layout-debugging spirals: don't judge fine layout from downscaled full-page shots, verify ground truth, stop after two failed fixes, and never degrade correct CSS to satisfy a screenshot. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Phrase the layout-verification rules at the capability level (viewport-height capture, computed-style measurement) with a thin 'In Studio' overlay naming the actual tools. Keeps the portable knowledge liftable into a shared, cross-harness skill while Studio still maps it to take_screenshot/measure_elements. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…rove-agent-visual-verification
1 task
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Related issues
How AI was used in this PR
Code, the new tool, the skill guidance, and this description were drafted with Claude Code and reviewed by me. The motivation came from watching a real
studio codebuild session spiral for ~24 minutes (164 turns) on a layout bug that did not exist; the changes here target that failure mode directly. Browser logic and the formatter were smoke-tested live.Proposed Changes
In a real build session, the agent spent ~24 minutes "fixing" a grid that was already correct. Root cause:
take_screenshotis always full-page, so a ~6000px-tall page becomes a ~273px-wide thumbnail after the vision pipeline downscales it — three columns blur into two, cards look "missing," and the agent chases phantom CSS bugs, eventually degrading correct CSS (repeat(3, 1fr)→auto-fit minmax(120px)) to make a blurry screenshot look different.This gives the agent reliable ways to see ground truth, and tells it when not to trust its eyes:
take_screenshotgainsfullPage: false: a crisp viewport-height slice (paged withoffset) that isn't downscaled, so fine layout detail stays legible.measure_elements— a new tool that returns how elements actually render (box + computedgrid-template-columns,flex-wrap,display, gaps, width, overflow) for a CSS selector. "Is it really 3 columns?" becomes a fact, not a guess.visual-designguidance — don't count columns/cards from a full-page shot; verify with a viewport shot ormeasure_elements; stop after two failed fixes of the same symptom; and never degrade real-browser-correct CSS to satisfy a screenshot.Why a Proof of Concept: the screenshot change is low-risk (full-page stays the default), but
measure_elementsis a new tool surface and the prompt/skill guidance is worth a team read for tone and placement. Open question: whether to makefullPage: falsethe default for in-build verification rather than opt-in.Testing Instructions
npm run cli:buildtake_screenshotfullPage: false+offset— the slice should be crisp enough to count columns. Compare to the full-page shot.measure_elementson the running site with a selector like.wp-block-columns— confirm it returns the renderedgrid-template-columns.npm test -- apps/cli/ai/tests/measure-elements.test.ts— unit tests for the result formatter (match / no-match / truncation).Pre-merge Checklist