Skip to content

Let take_screenshot skip chat display for internal verification captures - #4097

Merged
bcotrim merged 3 commits into
trunkfrom
add-take-screenshot-display-flag
Jul 7, 2026
Merged

Let take_screenshot skip chat display for internal verification captures#4097
bcotrim merged 3 commits into
trunkfrom
add-take-screenshot-display-flag

Conversation

@bcotrim

@bcotrim bcotrim commented Jul 6, 2026

Copy link
Copy Markdown
Contributor

Related issues

How AI was used in this PR

Implemented and tested with Claude Code, based on a confirmed finding from a multi-agent review of #3982; human-reviewed before opening.

Proposed Changes

Since #3982, every take_screenshot capture is automatically shown to the user as an inline chat artifact. That is right for deliberate captures, but the agent also screenshots constantly for its own eyes — the visual-polish loop alone takes up to 5 verification passes per page, and a light/dark × desktop/mobile check produces 4 images per call. Iterative design sessions flood the conversation with near-identical screenshots the user never asked for, and each one is persisted in the session transcript. (The previous presentation rule's "Do not present every internal verification screenshot" guidance was lost in the #3982 rewrite, and the model had no mechanism to comply with it anyway.)

This PR gives the agent that mechanism: an optional display input on take_screenshot (default true, so current behavior is unchanged). Passing display: false keeps the capture out of the chat while the model still receives the image for its own visual verification, and the presentation rule now tells the agent to use it while iterating so users only see deliberate milestone captures.

Testing Instructions

  • Run npm test -- apps/cli/ai/tests/tools.test.ts apps/cli/ai/tests/system-prompt.test.ts packages/common
  • Run npm run typecheck
  • Reproduce the flood on trunk: in Studio Code, prompt "Check the homepage for spacing and contrast issues in light and dark mode, on desktop and mobile. After every change you make, take screenshots again to verify. Keep iterating until it looks right." — the conversation fills with near-identical screenshots.
  • On this branch, repeat the same prompt: intermediate verification captures should stay out of the chat, with only deliberate/milestone captures displayed.
  • Ask directly for a screenshot ("take a screenshot of my site") and confirm it still appears inline as before.

Pre-merge Checklist

  • Have you checked for TypeScript, React or other console errors?

🤖 Generated with Claude Code

@wpmobilebot

wpmobilebot commented Jul 6, 2026

Copy link
Copy Markdown
Collaborator

📊 Performance Test Results

Comparing 087c1d5 vs trunk

app-size

Metric trunk 087c1d5 Diff Change
App Size (Mac) 1411.96 MB 1411.97 MB +0.00 MB ⚪ 0.0%

site-editor

Metric trunk 087c1d5 Diff Change
load 1106 ms 1082 ms 24 ms ⚪ 0.0%

site-startup

Metric trunk 087c1d5 Diff Change
siteCreation 6484 ms 6508 ms +24 ms ⚪ 0.0%
siteStartup 2389 ms 2389 ms 0 ms ⚪ 0.0%

Results are median values from multiple test runs.

Legend: 🟢 Improvement (faster) | 🔴 Regression (slower) | ⚪ No change (<50ms diff)

Comment thread package-lock.json Outdated
},
"bin": {
"pi-ai": "./dist/cli.js"
"pi-ai": "dist/cli.js"

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am not sure what are these changes so just leaving the comment to flag this 😅

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice catch! 🙌 Removed!

@katinthehatsite katinthehatsite left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The changes look good and I only saw the screenshots once the agent was done with the audit 👍

Image

I left one minor comment regarding some changes that did not seem to be related or I was not sure what they were

@bcotrim
bcotrim enabled auto-merge (squash) July 7, 2026 13:54
@shaunandrews

Copy link
Copy Markdown
Contributor

Weird. I don't have the same experience as y'all. The agent does take screenshots, but rarely more than one.

image

--

On this branch, I no longer see the screenshots. The whole point was to let users view the screenshots.

Its important to know what the agent is seeing, otherwise I can't trust its decision making.

For example, one of the sites I'm working on has some fancy JS on the homepage, but the agent (for some reason) can't see the output of that JS in it's screenshots. So, the agent works on false assumptions, and the user has no way to know that the agent is seeing incorrect output.

Perhaps we can still include the screenshots within the expanded tool call in the UI?

image
@katinthehatsite

Copy link
Copy Markdown
Contributor

On this branch, I no longer see the screenshots. The whole point was to let users view the screenshots.

Actually, now that I am thinking to when I was testing it, I had a mix of both: sometimes it showed Taking screenshots but did not show them to me and sometimes it did, but mostly at the end of the session when the agent was done working. I think this is consistent with what you are describing.

One solution might be to ensure that it always takes and displays screenshots at the beginning of the session and at the end of the session

@bcotrim
bcotrim merged commit 3eca460 into trunk Jul 7, 2026
11 checks passed
@bcotrim
bcotrim deleted the add-take-screenshot-display-flag branch July 7, 2026 14:21
shaunandrews added a commit that referenced this pull request Jul 10, 2026
Reconciles the July 6–10 trunk sync: adopts trunk's productionized
screenshot/theme tooling (#3982/#4097), grafts the copy-button (#4113),
edit-message (#4094), and inline chat artifacts into the redesigned
conversation views, keeps the merged Connector surface over trunk's
menu-action wiring (#4085), and keeps the blueprint-selector deletion.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

4 participants