Skip to content

Improve visual annotation workflows - #4411

Open
shaunandrews wants to merge 1 commit into
trunkfrom
iterate-annotation-workflows
Open

Improve visual annotation workflows#4411
shaunandrews wants to merge 1 commit into
trunkfrom
iterate-annotation-workflows

Conversation

@shaunandrews

@shaunandrews shaunandrews commented Jul 31, 2026

Copy link
Copy Markdown
Contributor

Related issues

How AI was used in this PR

Built through an iterative Codex session with continuous human UX feedback. Codex implemented the feature, hardened the data flow, and added tests; the interaction design was manually reviewed in an isolated Electron build.

Proposed Changes

  • Reworks Studio's preview annotation composer to match the main composer, with compact glass styling, clear cancel/save/send actions, keyboard shortcuts, dragging, scroll locking, scrim isolation, persistent selection outlines, and viewport-safe markers.
  • Keeps annotation mode active between notes and supports selecting deeply layered or overlapping elements without moving the composer.
  • Replaces the toolbar icon with clearer Annotate / Cancel / Send to chat controls and improves tooltip, spacing, and states.
  • Sends each annotation to chat as its own user message with an attached element chip containing useful tag, ID/class, and nearby-text context.
  • Persists structured annotation metadata end-to-end across optimistic UI state, desktop IPC, local/hosted connectors, CLI input payloads, and session replay.
  • Hardens Studio and standalone CLI annotation bridges with authenticated commands, payload validation, size/count limits, safer browser lifecycle handling, and bounded stored state.
  • Improves drag and layer-selection performance and adds accessibility semantics for dialogs, markers, and controls.

Review guide

Please focus on the injected preview inspector lifecycle, structured annotation persistence, and the security boundary between previewed pages and Studio. The styling and transcript rendering are comparatively self-contained.

⚠️ Visual change: needs human review in light + dark mode.

Testing Instructions

  1. Open a running site in Studio and click Annotate.
  2. Select normal and overlapping elements, cycle layers, drag the composer, and save several notes.
  3. Confirm selected/saved elements remain highlighted and edge markers stay visible.
  4. Use Return to save, Cmd/Ctrl+Return for a newline, then Send to chat.
  5. Confirm each note appears as a separate chat message with its element attachment.
  6. Run:
    • npm run typecheck
    • npm test -- apps/cli/ai/inspector/annotation-result.test.ts apps/cli/commands/ai/sessions/tests/resume.test.ts apps/cli/commands/ai/tests/index.test.ts apps/ui/src/components/site-preview/inspector-script.test.ts apps/ui/src/ui-classic/components/session-view/annotations.test.ts apps/ui/src/ui-classic/components/session-view/conversation/index.test.ts apps/ui/src/data/queries/use-agent-run.test.tsx apps/ui/src/data/core/connectors/local/index.test.ts packages/common/ai/inspector-annotations.test.ts packages/common/ai/visual-annotations.test.ts packages/common/ai/tests/chat-images.test.ts
    • npm run build --workspace @studio/ui

Focused result: 67 tests passed. The full repository suite was also attempted; unrelated environment-bound port, e2e, and color-output suites did not complete cleanly.

Pre-merge Checklist

  • Typecheck passes
  • Focused tests pass
  • Production UI build passes
  • Human visual review in light and dark mode
  • Full CI passes
@shaunandrews
shaunandrews marked this pull request as ready for review July 31, 2026 21:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

1 participant