Skip to content

Add /annotate skill with Studio inspector - #3243

Merged
youknowriad merged 3 commits into
trunkfrom
studio-inspector-annotate-skill
Apr 28, 2026
Merged

Add /annotate skill with Studio inspector#3243
youknowriad merged 3 commits into
trunkfrom
studio-inspector-annotate-skill

Conversation

@youknowriad

Copy link
Copy Markdown
Contributor

Related issues

  • Builds on Add visual annotation browser with Agentation #3030 (the original /annotate skill PR by @lezama). This PR rebases that work onto current trunk and replaces the third-party agentation toolbar with a self-contained Studio inspector. The first commit on this branch is the original @lezama commit, preserved for attribution; the second commit is the rewrite.

How AI was used in this PR

Heavy AI involvement — Claude Code drove the rebase, the discovery of the issues with the agentation-based approach (window sizing, dock-icon leak, sync gaps, missing confirmation step), and the rewrite into a vanilla-DOM inspector. End-to-end behaviours were verified with Playwright scripts; screenshots from each iteration are in the session transcript.

Reviewers should focus on:

  • The injected page script in apps/cli/ai/inspector/page-script.ts — it runs in the host WordPress page so it needs to be defensive against any theme CSS / global handlers.
  • The chromium lifecycle in apps/cli/ai/inspector/inspector-inject.ts — close-on-page-close + close-on-process-exit hooks are the bits I'd most want a second pair of eyes on.
  • The mandatory "summarize and confirm" step in SKILL.md. The intent is that the agent never edits files based on annotations without an explicit yes from the user.

Proposed Changes

This PR is a Proof-of-Concept-grade rework of the annotation skill. Marking as draft + Proof of Concept label per AGENTS.md guidance for substantial AI-generated features.

Removed

Added — apps/cli/ai/inspector/

  • page-script.ts — self-contained vanilla-DOM inspector mounted into a Shadow DOM root. No React, no esm.sh, no external runtime deps.
  • inspector-inject.ts — Playwright launch + inject + waitForAnnotationsDone helper.

Inspector UI (vs. agentation toolbar in #3030)

  • Two-button toolbar: Annotate and Done, plus a count badge. Layout-mode, settings page, theme picker, colour palette all gone.
  • Annotate is sticky — after Save, picking mode stays on so the user can chain annotations.
  • Done ships the batch to the CLI via window.__studioAnnotateDone, clears local state, plays a live 10-second countdown toast, then the CLI auto-closes the chromium window.
  • Picked elements get a blue outline; saved annotations get numbered markers; clicking a marker reopens the popup for edit/delete.
  • Esc cancels picking / closes the popup.

CLI lifecycle hardening

  • 1280×800 window + viewport: null so the bottom-right toolbar is always inside the visible region regardless of host screen size.
  • bringToFront() so the macOS window doesn't open behind the terminal.
  • page.on('close') and browser.on('disconnected') tear down the whole chromium process tree.
  • process.once('exit'|'SIGINT'|'SIGTERM'|'SIGHUP') hooks so quitting studio ai doesn't orphan a dock icon.
  • 10-second deferred close after Done so the user has time to register the confirmation before the window vanishes.

Skill flow (SKILL.md)

  • After wait_for_annotations returns, the agent must print a numbered summary in plain language and call AskUserQuestion to confirm before making any edits.
  • Each summary item identifies the element by tag + nearbyText (not selector), restates the intent in the agent's own words, and quotes the user's verbatim comment.
  • For another annotation round, the user re-runs /annotate rather than reusing the (now-closed) window.

Testing Instructions

  1. npm install && npm run cli:build
  2. node apps/cli/dist/cli/main.mjs ai
  3. In the chat, run /annotate against any local Studio site.
  4. Verify the bottom-right Annotate / Done toolbar appears in the headed chromium window. Click Annotate, click any element on the page, type feedback, click Save. Picking mode should stay on — click another element and repeat without re-clicking Annotate.
  5. Click Done. Confirm:
    • A toast counts down 10s … 1s … closing now….
    • The chromium window closes automatically.
    • The CLI prints a numbered summary (in plain language, with the verbatim comment in parens) and asks for confirmation before making any edits.
  6. Confirm the changes get applied. Try /annotate again — a fresh window should open with no leftover annotations from the previous session.
  7. Close the chromium window manually mid-session — the dock icon should disappear cleanly (no force-quit needed).
  8. Quit studio ai (Ctrl-C) while the inspector window is open — the chromium process should exit too.

Pre-merge Checklist

  • No TypeScript / lint errors (npm run typecheck --workspace apps/cli, npx eslint --fix on the modified files).
  • CLI builds (npm run cli:build).
  • Unit/E2E tests — none added in this PR. The interaction surface lives entirely inside an injected vanilla-DOM script in a headed chromium window; reviewers may want to push for Playwright coverage before landing.
  • Cross-platform (Windows/Linux) verification — only manually tested on macOS.
  • PolyForm-Shield licence concerns from the agentation package are gone, but reviewers should confirm there are no other lingering references.
@youknowriad
youknowriad marked this pull request as ready for review April 25, 2026 18:37
*
* Hand-off to the agent: clicking "Done" sets `window.__studioAnnotateDone`,
* which `waitForAnnotationsDone()` in inspector-inject.ts polls for.
*/

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If this one becomes more complex we could consider extracting it to its own "react app" in the apps folder.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good shout — agreed. Right now the inspector is ~330 lines of vanilla DOM in a Shadow root, which is light enough to live as an injected string. If we start adding things like keyboard navigation, an annotations panel, or theme variants, hoisting it to a proper apps/inspector React app (built and shipped as a static bundle the CLI loads as a string) would be the natural next step.

lezama and others added 2 commits April 27, 2026 16:19
Drops the dependency on the agentation npm package + agentation-mcp
server in favour of a vanilla-DOM inspector injected from the CLI.
The original PR #3030 wired up the agentation library, but in practice
a series of issues surfaced that all stem from depending on a
third-party React component loaded from esm.sh:

  - Window-sizing mismatch hid the toolbar bottom-right corner on 13"
    MacBooks (1440×900 page viewport vs. window-with-chrome).
  - Closing the page didn't tear down chromium, leaving a phantom
    dock icon until force-quit.
  - The agentation toolbar showed layout-mode and settings buttons
    we don't need; hiding them required CSS :has() hacks against
    hash-suffixed class names.
  - Annotations only synced from page localStorage to the MCP
    server's SQLite DB on toolbar interaction, so the CLI saw "0
    annotations" while the page badge showed a count.
  - There was no clear "I'm done" hand-off; the agent had to call
    agentation MCP tools and hope the server was caught up.

The replacement is a single TypeScript file (page-script.ts) that
exports a self-contained vanilla-DOM widget mounted into a Shadow
DOM root. No React, no esm.sh, no external runtime deps. All page-CSS
isolation is handled by the shadow root rather than CSS-modules tricks.

UX changes vs. the agentation toolbar:

  - Two-button toolbar: "Annotate" and "Done", plus a count badge.
    Layout-mode, settings page, theme picker, color palette all gone.
  - Annotate is sticky — after Save, picking mode stays on so the
    user can chain annotations without re-clicking the button.
  - Done sends the batch to the CLI via window.__studioAnnotateDone,
    clears local state, and triggers an auto-close with a live 10s
    countdown toast so the user sees what's happening.
  - Picked elements get a blue outline; saved annotations get
    numbered markers; clicking a marker reopens the popup for
    edit/delete.
  - Esc cancels picking / closes the popup.

CLI side (inspector-inject.ts):

  - Pinned 1280×800 window + viewport: null so the bottom-right
    toolbar is always inside the visible region regardless of host
    screen size.
  - bringToFront() so the macOS chromium window doesn't open behind
    the terminal.
  - page.on('close') and browser.on('disconnected') tear the whole
    chromium process tree down, plus an exit/SIGINT/SIGTERM/SIGHUP
    hook so quitting `studio ai` doesn't orphan a dock icon.
  - 10-second deferred close after Done so the page-side countdown
    has time to play out and the user can register the confirmation
    before the window vanishes.

Skill flow (SKILL.md):

  - After wait_for_annotations returns, the agent is required to
    print a numbered summary in plain language and ask the user to
    confirm via AskUserQuestion BEFORE making any changes.
  - Each summary item identifies the element by tag + nearbyText
    (not selector), restates intent in the agent's own words, and
    quotes the user's verbatim comment.
  - For another round, the user re-runs `/annotate` rather than
    re-using the closed window.

Removed:
  - agentation-mcp dev dependency
  - agentation MCP server entry from agent.ts
  - apps/cli/ai/agentation-inject.ts
@youknowriad
youknowriad force-pushed the studio-inspector-annotate-skill branch from c41127b to 868bc16 Compare April 27, 2026 15:21
@wpmobilebot

wpmobilebot commented Apr 27, 2026

Copy link
Copy Markdown
Collaborator

📊 Performance Test Results

Comparing 99d7d56 vs trunk

app-size

Metric trunk 99d7d56 Diff Change
App Size (Mac) 1451.96 MB 1907.46 MB +455.49 MB 🔴 31.4%

site-editor

Metric trunk 99d7d56 Diff Change
load 1795 ms 1786 ms 9 ms ⚪ 0.0%

site-startup

Metric trunk 99d7d56 Diff Change
siteCreation 8086 ms 8083 ms 3 ms ⚪ 0.0%
siteStartup 4939 ms 4949 ms +10 ms ⚪ 0.0%

Results are median values from multiple test runs.

Legend: 🟢 Improvement (faster) | 🔴 Regression (slower) | ⚪ No change (<50ms diff)

@borkweb borkweb left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A couple of small changes

Comment thread apps/cli/ai/inspector/page-script.ts
Comment thread apps/cli/ai/tools.ts
- Exclude the inspector host element when counting same-tag siblings in
  buildSelector. Without this, our injected #__studio-inspector-host
  appears in parent.children and can cause an off-by-one nth-of-type
  index for top-level body div children. Conceptually wrong even when
  the index happens to align — we shouldn't count our own injected
  scaffolding as part of the user's site structure.

  Per @borkweb's review on PR #3243.

- Constrain `wait_for_annotations` timeoutMinutes to 1..120 (integer).
  Previously z.number().optional() allowed 0, which would feed
  Playwright's `timeout: 0` (= no timeout, block forever) and lock the
  agent up. 120 minutes is generous enough for any realistic
  annotation session.

  Per @borkweb's review on PR #3243.
@youknowriad
youknowriad merged commit 37ff001 into trunk Apr 28, 2026
16 checks passed
@youknowriad
youknowriad deleted the studio-inspector-annotate-skill branch April 28, 2026 10:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

4 participants