Add Jetpack catch-all rule for non-core feature requests

[lezama]

Related issues

• Follow-up to Steer agent to use Jetpack Forms instead of raw HTML forms #3351, which introduced the PLUGIN_RECOMMENDATIONS registry and shipped the first specific rule (Jetpack Forms).
• Driven by a discussion on the Steer agent to use Jetpack Forms instead of raw HTML forms #3351 thread where Riad and I agreed to "try and see" whether a generic Jetpack catch-all helps the agent default to Jetpack blocks when no core block fits.

How AI was used in this PR

Code change drafted by Claude after we sketched the rule in conversation. Opening as a draft because the design point ("specific rules vs. generic catch-all") isn't fully settled — Riad's preference is specific entries (form → Jetpack Forms, slider → Jetpack Slider, …), and this catch-all is offered as a complement to them, not a replacement. Claude also drafted this PR description.

Proposed Changes

Adds one new entry to PLUGIN_RECOMMENDATIONS in apps/cli/ai/plugin-recommendations.ts:

• name: 'Jetpack (catch-all)', no htmlPatterns — this rule only contributes guidance text to the system prompt; it does not enforce policy.
• Steers the agent: when no core block cleanly fits a request (sliders, social icons, related-posts grids, business-hours blocks, etc.), check Jetpack first before reaching for raw HTML in core/html.
• Tells the agent how to discover the available Jetpack block list at runtime (wp_cli eval against WP_Block_Type_Registry) and how to enable a module if the expected block isn't registered (wp_cli jetpack module list / activate).
• Notes that the specific rules above (Forms today, more later) take precedence — the catch-all is the last stop before core/html.

No structural changes to PluginRecommendation (no new fields). The current free-form guidance prose is enough; following @youknowriad's existing design, structure is added when an automated loop needs it (htmlPatterns for the policy loop), not preemptively.

Why now / why this shape

• Empirically (from the dotcom-studio Slack discussion and a session yesterday), the agent's failure mode in this area isn't "wrong syntax" — it's "didn't think of Jetpack." A short prompt nudge is the cheapest thing that can move the needle.
• One free-form entry costs almost nothing; if specific rules (slider, gallery, social-icons) replace it later, ripping it out is one delete.
• The discovery step (wp_cli eval … WP_Block_Type_Registry) sidesteps the fact that Jetpack's block availability depends on which modules are active. Hard-coding a list would go stale; letting the agent introspect at runtime stays accurate.

Testing Instructions

The catch-all is steering, not policy enforcement, so its effect is observable through agent behavior, not unit tests:

1. Check out this branch.
2. npm run cli:build
3. Create a fresh local site: node apps/cli/dist/cli/main.mjs site create --path ~/Studio/jetpack-catchall-test --name jetpack-catchall-test --skip-browser
4. From that site directory, run a prompt that has no clean core-block solution. Examples:
   • studio ai "add a slideshow of 3 placeholder images to the homepage"
   • studio ai "add a row of social icons (twitter, github, linkedin) to the footer"
   • studio ai "add a related-posts grid below the main content"
5. Expected: the agent installs Jetpack (if not already), runs the discovery wp_cli eval to see what's registered, activates a module if the block it wants is gated, and uses the Jetpack block. It should NOT produce a core/html block with raw <div class="slideshow">…</div> markup.
6. Existing unit tests still pass: cd apps/cli && npx vitest run ai/tests/block-content-policy.test.ts.

I haven't run the end-to-end prompts above — leaving that for the reviewer to decide whether to validate manually or merge this on architecture/code-review alone.

Pre-merge Checklist

• Have you checked for TypeScript, React or other console errors?

[youknowriad]

social icon exist in core.

[lezama]

Good catch — core/social-links covers that. Dropped social icons from the example list in 4891b42a.

[youknowriad]

Can we add an eval for this, how do I test this?

[lezama]

Added a promptfoo eval in `4891b42a`: `jetpack-catchall-slideshow` (in `eval/promptfoo.config.yaml`).

It tells the agent "create a site, add a slideshow of three placeholder images" — there's no clean core block for slideshow, so the catch-all should fire. Two assertions:

1. Agent ran `wp_cli plugin install jetpack` while fulfilling the request.
2. The page content the agent generated contains a `` block, and does not contain a `core/html` block with raw `
   ` / carousel markup.

To run just this test:

```sh
studio auth login # if you aren't already
npm run cli:build
npx promptfoo@0.121.4 eval -c eval/promptfoo.config.yaml --filter-pattern "catch-all"
```

I ran it on this branch — passes (1/1 in 2m 52s, agent installed Jetpack and used `jetpack/slideshow`). The eval is also listed in `eval/README.md`.

If the assertion shape is wrong (too lenient / too strict / not the signal you want measured) happy to iterate on it.

[lezama]

Ran a /simplify pass — pushed ceb2c211. Net −26 lines.

Applied:

1. Renamed the entry Jetpack (catch-all) → Jetpack (general) (parallel to Jetpack Forms, drops the jargon).
2. Made blocks?: string[] optional in the PluginRecommendation interface and dropped blocks: [] from the new entry. The empty array was semantically wrong — Jetpack registers many blocks; the entry just doesn't enumerate them. The runtime-discovery flow (WP_Block_Type_Registry) is the source of truth here, so leaving the field off is more accurate than [].
3. Collapsed the eval test from 2 assertions to 1. The dropped assertions were:
   • "agent installed jetpack" — strictly implied by the surviving content check (a wp:jetpack/* block in the page implies the plugin is installed). The diagnostic split wasn't worth the duplication.
   • "no raw slideshow HTML in core/html" — fragile (slid|carousel) regex that adds little once the positive usedJetpackBlock check holds.
4. Trimmed the "Catch-all signal #N" narration comments. The file's existing convention (e.g. the --post_content-file= regression block) uses comments for why an assertion exists, not what it does — the surviving assertion is short enough to read directly.

Flagged out of scope (separate PR if you want it): the marker-extraction boilerplate (output.split → find EVAL_RUNNER_RESULT_FILE → readFileSync → JSON.parse) is duplicated across 8 assertions in this file. PromptFoo's defaultTest.options.transform (or a shared eval/result-loader.mjs) could centralize it, but that touches every existing assertion and feels like its own cleanup.

[youknowriad]

I didn't test but if you did land this, it looks good to me.

[wpmobilebot]

📊 Performance Test Results

Comparing c123d14 vs trunk

app-size

Metric	trunk	c123d14	Diff	Change
App Size (Mac)	1410.40 MB	1410.40 MB	+0.00 MB	⚪ 0.0%

site-editor

Metric	trunk	c123d14	Diff	Change
load	1512 ms	1502 ms	10 ms	⚪ 0.0%

site-startup

Metric	trunk	c123d14	Diff	Change
siteCreation	8089 ms	8091 ms	+2 ms	⚪ 0.0%
siteStartup	4921 ms	4915 ms	6 ms	⚪ 0.0%

---

Results are median values from multiple test runs.

Legend: 🟢 Improvement (faster) | 🔴 Regression (slower) | ⚪ No change (<50ms diff)

[lezama]

I need an approval :)

[youknowriad]

By pass approvals :P It's RSM

[youknowriad]

Over time comes blocks might get hidden from the inserter but kept for BC, we probably should exclude the inserter hidden blocks from this.

[lezama]

Follow up #3389