Add CLI integration test for site create (custom name, domain, HTTPS) by gavande1 · Pull Request #3947 · Automattic/studio

gavande1 · 2026-06-24T12:06:21Z

Related issues

Related to STU-1867

How AI was used in this PR

Proposed Changes

The end-to-end suite drives every site action through the Studio UI, which is slow, flaky, and ties coverage to the UI. This adds the first real CLI integration test for studio site create: it runs the actual command against an isolated config directory and asserts on the persisted result, covering a site with a custom name and a site with a custom domain plus HTTPS. The tests are tagged e2e so they run in the slower release or manual suite rather than on every pull request.

Testing Instructions

Build the CLI with npm run cli:build.
Run npm test -- --tagsFilter='e2e' and confirm both tests pass.
Run npm test -- --tagsFilter='!e2e' and confirm these tests are skipped, so the fast suite is unaffected.

Pre-merge Checklist

Have you checked for TypeScript, React or other console errors?

gcsecsey

Nice work @gavande1, spawning the CLI and asserting against files on disk is a good direction for these tests IMO.

The two tests cases added here LGTM, and we can add more incrementally as needed.

wpmobilebot · 2026-06-26T07:18:36Z

📊 Performance Test Results

Comparing ed958ee vs trunk

app-size

Metric	trunk	`ed958ee`	Diff	Change
App Size (Mac)	1315.48 MB	1315.48 MB	+0.00 MB	⚪ 0.0%

site-editor

Metric	trunk	`ed958ee`	Diff	Change
load	1047 ms	1043 ms	4 ms	⚪ 0.0%

site-startup

Metric	trunk	`ed958ee`	Diff	Change
siteCreation	6479 ms	6469 ms	10 ms	⚪ 0.0%
siteStartup	7005 ms	6983 ms	22 ms	⚪ 0.0%

Results are median values from multiple test runs.

Legend: 🟢 Improvement (faster) | 🔴 Regression (slower) | ⚪ No change (<50ms diff)

…3961) ## Related issues - Fixes STU-1871 - Stacked on #3947 ## How AI was used in this PR I used Claude to review the existing Playwright blueprint tests and the new CLI e2e harness from #3947, plan the migration, and to write the test. I reviewd and verified all changes. ## Proposed Changes Blueprint coverage currently lives only in `apps/studio/e2e/blueprints.test.ts`, which boots the entire desktop app through Playwright to create a site from a custom Blueprint and then inspects wp-admin — slow, flaky, and coupled to the UI. This PR continues the "Migrate Studio E2E tests to CLI" effort by exercising the same capability directly through the CLI. - Adds integration tests that run `studio site create --blueprint <file>` against an isolated config directory and asserts on the file system. - Closes a gap in the #3947 harness: the helper isolated the config directory but not the legacy Electron appdata directory, so the Studio-compatibility startup migration could find a real pre-split `appdata-v1.json` and abort the CLI on a developer's machine. The harness now also isolates that path, which fixes the sibling `create.e2e.test.ts` locally. ## Testing Instructions 1. Build the CLI: `npm run cli:build`. 2. Run the new suite (needs a network connection for the download install step): `npm test -- apps/cli/commands/site/tests/create-blueprint.e2e.test.ts --tagsFilter='e2e'` Confirm all four tests pass. 3. Confirm the fast suite is unaffected: `npm test -- apps/cli/commands/site/tests/create-blueprint.e2e.test.ts --tagsFilter='!e2e'` The four tests are skipped. 4. Regression check for the harness fix — confirm the sibling test also passes: `npm test -- apps/cli/commands/site/tests/create.e2e.test.ts --tagsFilter='e2e'` ## Pre-merge Checklist - [x] Have you checked for TypeScript, React or other console errors? --------- Co-authored-by: Rahul Gavande <rahul.gavande@automattic.com>

fredrikekelund

I left some comments about doing HTTP requests to the site's homepage after running the site create commands. The fundamentals here look really good, though 👍

fredrikekelund · 2026-06-29T13:37:48Z

+		expect( result.code, result.stderr ).toBe( 0 );
+
+		// The site is persisted to the real cli.json with the custom name.
+		const config = readCliConfig( env );
+		expect( config.sites ).toHaveLength( 1 );
+		const [ site ] = config.sites;
+		expect( site.name ).toBe( siteName );
+		expect( site.path ).toBe( sitePath );
+		expect( site.phpVersion ).toBeTruthy();
+		expect( site.running ).toBe( false );
+
+		// Real WordPress core files were copied into the site directory.
+		// (wp-config.php is generated at server start, which --no-start skips.)
+		expect( fs.existsSync( path.join( sitePath, 'wp-load.php' ) ) ).toBe( true );
+		expect( fs.existsSync( path.join( sitePath, 'wp-includes', 'version.php' ) ) ).toBe( true );


I'd add an HTTP request that verifies that we get a 200 OK HTTP response from the site. Either we need to parse the regular stdout output from the site create command, or we add a --format=json parameter (that's the kind of change I was talking about in our huddle where working on the test suite might uncover flaws in the CLI implementation)

Done in #4000. The create test now starts the site and asserts a 200 with HTML from the homepage.

fredrikekelund · 2026-06-29T13:38:53Z

+	type CliEnv,
+} from './helpers/cli-e2e';
+
+describe.skipIf( ! cliE2ePrerequisitesMet() )( 'CLI e2e: studio site create', () => {


It seems more helpful if the test would break rather than gracefully skip if the CLI artifacts don't exist.

We already have open issue to handle CLI tests.

fredrikekelund · 2026-06-29T13:39:13Z

+			expect( result.code, result.stderr ).toBe( 0 );
+
+			// The custom domain and HTTPS preference are persisted to cli.json.
+			// (--no-start skips the hosts-file / certificate setup that running would do.)
+			const config = readCliConfig( env );
+			expect( config.sites ).toHaveLength( 1 );
+			const [ site ] = config.sites;
+			expect( site.name ).toBe( siteName );
+			expect( site.customDomain ).toBe( customDomain );
+			expect( site.enableHttps ).toBe( true );
+			expect( site.running ).toBe( false );
+
+			expect( fs.existsSync( path.join( sitePath, 'wp-load.php' ) ) ).toBe( true );


Same thing here about adding an HTTP request for verification

I kept the custom-domain and HTTPS case as persistence-only. Serving it needs the full stack: the proxy on ports 80 and 443, a trusted root CA, and a write to the shared /etc/hosts behind an elevated-privileges prompt, so it cannot run in an isolated, headless test. The live check stays on the plain localhost site in #4000.

fredrikekelund · 2026-06-29T13:39:39Z

+			version: 1,
+			sites: [],
+			snapshots: [],
+			lastDependencyCheckTime: Date.now(),


gavande1 · 2026-06-30T08:37:52Z

Thank you for the review @fredrikekelund. I have addressed your comments in #4000.

## Related issues - Fixes STU-1868 - Builds on #3947 (the `site create` integration test + e2e harness) and complements the UI suite in #3950 ## How AI was used in this PR I used Claude to write the test and the harness changes, and traced the daemon connection failures to a Unix socket path length limit. I reviewed the diff, ran the suite on nodes 22 and 24, and confirmed that runs are isolated, with no orphan daemons/servers and no temp-dir leaks. ## Proposed Changes We're migrating the legacy blueprint-based start/stop test cases to the CLI integration approach. The existing `start.test.ts` / `stop.test.ts` unit tests mock the daemon and server manager. This PR instead boots a real WordPress server through the process-manager daemon and verifies the live running state via `site list --format json`, so it actually exercises the daemon -> server -> port lifecycle. - Adds an `e2e`-tagged suite that creates a site with `--no-start`. It runs in the slower (release/manual) suite, not on every PR. - Updates the harness to prevent the tests affecting local sites. The lifecycle commands talk to the process-manager daemon, and the cleanup step `stop --all` would kill the developer's real running sites. The harness now gives each run its own daemon home via `STUDIO_PROCESS_MANAGER_HOME`, so runs are fully isolated and parallel-safe. That home is deliberately kept short and under the temp dir: the daemon's control socket is a Unix domain socket, and nesting it under the `studio-cli-e2e-<uuid>` root path overflows macOS's 104-char socket path limit. This surfaces as `EINVAL` on node 24 and as an indefinite connect-retry (hang) on node 22. Windows uses a fixed named pipe, so this isolation is macOS/Linux only. - The harness also seeds a minimal `app.json` so the `00-check-studio-compatibility` migration returns early. Without it, that migration falls through to the developer's real legacy app data and aborts the CLI, so the suite would pass on CI yet fail on a dev machine with Studio installed. This also retroactively hardens the `site create` test from #3947, which carries the same gap. - Uses `--runtime sandbox` to stay hermetic. The native PHP runtime drives the same lifecycle but downloads its ~25 MB binary into the isolated config dir on the first run. Covering it hermetically needs CI to provision that binary, so native-PHP coverage is a follow-up. ## Testing Instructions 1. `npm run cli:build` 2. `npm test -- --tagsFilter='e2e' apps/cli/commands/site/tests/start-stop.e2e.test.ts` 3. Expect 2 tests green (`starts a site`, `stops a site`). 4. While it runs, confirm it does not disturb sites you have running in Studio, and leaves no orphan processes or temp dirs afterward. ## Pre-merge Checklist - [x] Have you checked for TypeScript, React or other console errors?

…te suite (covered by CLI #3947)

## Related issues - Related to STU-1867 ## How AI was used in this PR Claude was used to write the code and explore the solution. The author reviewed it manually. ## Proposed Changes Follow-up to the CLI e2e tests in #3947. The `site create` test only asserted persisted state on disk. This adds a real check: after creating and starting a site, the test requests its homepage and confirms a 200 with HTML, so it now fails if a created site cannot be served. This does not cover the custom-domain and HTTPS case, which cannot run in an isolated, headless test. See the discussion in #3947. ## Testing Instructions 1. Build the CLI with `npm run cli:build`. 2. Run `npm test -- apps/cli/commands/site/tests/create.e2e.test.ts --tagsFilter='e2e'` and confirm both tests pass. 3. Run the same with `--tagsFilter='!e2e'` and confirm the tests are skipped, so the fast suite is unaffected. ## Pre-merge Checklist - [x] Have you checked for TypeScript, React or other console errors?

## Related issues - Related to STU-1867 - companion to #3947 ## How AI was used in this PR Claude audited the existing renderer tests to find coverage gaps, and wrote the tests. ## Proposed Changes The Site Management e2e tests drive every action through the UI, which is slow, flaky, and couples coverage to the current renderer. As we discussed on the huddle, durable logic coverage will live in the CLI integration suite (#3947), which is UI-independent. This PR lands a small set of UI IPC-mock tests as reference examples for the command-boundary pattern on the current renderer. These tests mount the real components and `SiteDetailsProvider`, mock only the IPC bridge, and assert the commands each user action generates. - ~~Covers all nine Site Management [smoke-sheet rows](https://docs.google.com/spreadsheets/d/1qpxqthl2wTPjrLEv9B-ckmvnIKJGn6kqoL3LrmrDAkk/edit?gid=1768483022#gid=1768483022), plus custom domain and HTTPS.~~ The tests now only cover duplicate, delete, rename, and PHP version. - Adds coverage for site duplication, since there is no `studio site duplicate` command for the CLI suite to test. - The delete cases now run on every platform (the old e2e versions were skipped on Windows). - These supersede the hook-boundary assertions in the sibling unit tests, which can be retired in a follow-up. The existing real e2e suite (apps/studio/e2e/sites.test.ts) is untouched and keeps running on every OS for true app→CLI and cross-platform coverage. These will be retired only once the new UI and the CLI round-trips are green. ## Testing Instructions - Run the new tests: ``` npm test -- src/components/tests/content-tab-settings.actions.test.tsx src/modules/site-settings/tests/edit-site-details.actions.test.tsx ``` - Confirm all tests pass - Run `npm run typecheck` and confirm there are no errors ## Regression validation I confirmed that these tests fail on a real regression and that they catch the same bug as the existing e2e suite. I made this change ran both suites: ```diff diff --git a/apps/studio/src/hooks/use-delete-site.ts b/apps/studio/src/hooks/use-delete-site.ts index 7cc117a..33055caa8 100644 --- a/apps/studio/src/hooks/use-delete-site.ts +++ b/apps/studio/src/hooks/use-delete-site.ts @@ -33,7 +33,7 @@ export function useDeleteSite() { if ( response === DELETE_BUTTON_INDEX ) { try { - await deleteSite( siteId, checkboxChecked ); + await deleteSite( siteId, ! checkboxChecked ); } catch ( error ) { ipcApi.showErrorMessageBox( { title: __( 'Deletion failed' ), EOF ``` | Suite | Bug introduced (inverted delete-files flag) | | --- | --- | | Old e2e (`sites.test.ts`) | <img width="1626" height="1552" alt="CleanShot 2026-06-25 at 11 28 34@2x" src="https://github.com/user-attachments/assets/1782f468-f435-4119-b03c-691cba1b8663" /> | | New UI integration tests | <img width="1640" height="1250" alt="CleanShot 2026-06-25 at 11 10 14@2x" src="https://github.com/user-attachments/assets/6683e5f8-685f-40da-a10a-cb1107b63bf3" /> | The existing unit tests mock the hook→IPC translation layer, so these aren't catching the introduced issue, only the existing e2e tests and the newly added integration tests are. ## Pre-merge Checklist - [x] Have you checked for TypeScript, React or other console errors? --------- Co-authored-by: Rahul Gavande <rahul.gavande@automattic.com>

Add CLI integration test for site create (custom name, domain, HTTPS)

ba829de

github-actions Bot assigned gavande1 Jun 24, 2026

gcsecsey mentioned this pull request Jun 24, 2026

Add Site Management UI integration tests #3950

Merged

1 task

Merge branch 'trunk' into stu-1867-add-cli-create-e2e-test

78450f3

gcsecsey marked this pull request as ready for review June 25, 2026 11:35

gcsecsey requested a review from fredrikekelund June 25, 2026 11:35

gcsecsey approved these changes Jun 25, 2026

View reviewed changes

Comment thread apps/cli/commands/site/tests/helpers/cli-e2e.ts

gcsecsey mentioned this pull request Jun 25, 2026

Add CLI integration test for creating a site from a custom Blueprint #3961

Merged

1 task

gavande1 enabled auto-merge (squash) June 26, 2026 06:59

Merge branch 'trunk' into stu-1867-add-cli-create-e2e-test

ed958ee

gavande1 disabled auto-merge June 26, 2026 07:01

gavande1 enabled auto-merge (squash) June 26, 2026 07:03

gavande1 merged commit 21f7d1a into trunk Jun 26, 2026
11 checks passed

gavande1 deleted the stu-1867-add-cli-create-e2e-test branch June 26, 2026 07:18

gcsecsey mentioned this pull request Jun 26, 2026

Add CLI integration test for site start/stop lifecycle #3977

Merged

1 task

fredrikekelund reviewed Jun 29, 2026

View reviewed changes

gavande1 mentioned this pull request Jun 30, 2026

CLI test: add HTTP response check for created site #4000

Merged

1 task

gcsecsey added a commit that referenced this pull request Jul 1, 2026

Reduce Site Management UI tests to reference examples; drop create-si…

544ae5a

…te suite (covered by CLI #3947)

gcsecsey mentioned this pull request Jul 2, 2026

DO NOT MERGE: Revert CLI e2e tests to check Windows E2E runtime #4056

Closed

1 task

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Add CLI integration test for site create (custom name, domain, HTTPS) - #3947

Add CLI integration test for site create (custom name, domain, HTTPS)#3947
gavande1 merged 3 commits into
trunkfrom
stu-1867-add-cli-create-e2e-test

gavande1 commented Jun 24, 2026

gcsecsey left a comment

Uh oh!

wpmobilebot commented Jun 26, 2026

Uh oh!

fredrikekelund left a comment

fredrikekelund Jun 29, 2026

gavande1 Jun 30, 2026

fredrikekelund Jun 29, 2026

gavande1 Jun 30, 2026

fredrikekelund Jun 29, 2026

gavande1 Jun 30, 2026

fredrikekelund Jun 29, 2026

gavande1 commented Jun 30, 2026 •

edited

Loading

Labels

4 participants

Uh oh!

Conversation

gavande1 commented Jun 24, 2026

Related issues

How AI was used in this PR

Proposed Changes

Testing Instructions

Pre-merge Checklist

gcsecsey left a comment

Choose a reason for hiding this comment

Uh oh!

wpmobilebot commented Jun 26, 2026

📊 Performance Test Results

app-size

site-editor

site-startup

Uh oh!

fredrikekelund left a comment

Choose a reason for hiding this comment

fredrikekelund Jun 29, 2026

Choose a reason for hiding this comment

gavande1 Jun 30, 2026

Choose a reason for hiding this comment

fredrikekelund Jun 29, 2026

Choose a reason for hiding this comment

gavande1 Jun 30, 2026

Choose a reason for hiding this comment

fredrikekelund Jun 29, 2026

Choose a reason for hiding this comment

gavande1 Jun 30, 2026

Choose a reason for hiding this comment

fredrikekelund Jun 29, 2026

Choose a reason for hiding this comment

gavande1 commented Jun 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Labels

4 participants

gavande1 commented Jun 30, 2026 •

edited

Loading