Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
31 changes: 31 additions & 0 deletions apps/cli/ai/skills/liberate/SKILL.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,31 @@
---
name: liberate
description: Import and rebuild a website from a closed platform (Wix, Squarespace, Webflow, Shopify, GoDaddy, Hostinger, HubSpot, Weebly) into a Studio WordPress site. Extracts pages/posts/products + media, then reconstructs the design as editable blocks + WooCommerce OR as a high-fidelity replica theme. Invoke when the user wants to migrate, import, liberate, or rebuild a site from one of these platforms.
---

# Liberate a website into Studio

This skill is only a **redirect**. The real, always-up-to-date pipeline lives in the Data Liberation engine's own skill. Do NOT re-plan or summarize the steps here — defer to the engine skill so its updates take effect automatically. Your job is just to (1) stand the engine up and (2) follow its skill, translating its tool calls into Studio's bridge.

## Step 1 — Locate the engine

Call the `data_liberation` tool with `{ tool: "setup" }`. It returns `{ engineDir, liberateSkill, skillsDir }` — the paths to the engine's skill files. The engine ships prebuilt with Studio, so this is instant — just proceed.

## Step 2 — Load the engine's tool catalog

Call `data_liberation` with `{ tool: "list" }` to load the engine's full tool catalog (every tool name + its argument JSON-Schema). Treat that catalog as the **source of truth** for tool names and arguments — the engine skill names the tools to call, but the catalog tells you their exact arguments. If you are ever unsure of an argument, re-read the catalog rather than guessing.

## Step 3 — Follow the engine's real liberate skill

`Read` the `liberateSkill` path from Step 1 (the engine's own `SKILL.md`) and execute its workflow verbatim, applying these Studio bindings wherever it instructs you:

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a very weird way to use the data_liberation. If the data liberation packages bundles skills, can't we actually "import" them directly where we define the skills of Studio Code instead of having a "wrapper" skill?


- **Engine tool call** — where it says "call the `liberate_X` tool with `{…}`", instead call Studio's `data_liberation` tool with `{ tool: "liberate_X", args: {…} }` and read the returned text/JSON as that tool's result. (`args` is a JSON **object**, never a stringified JSON.)
- **Sub-skill dispatch** — where it dispatches another skill (e.g. `replicate-with-blocks` / `replicate-theme`), `Read` it under `engineDir/skills/<name>/SKILL.md` and continue inline with these same bindings.
- **Install & import** — follow the engine's install step (it documents both `liberate_preview` and `liberate_install_theme` + `liberate_import`); `/liberate` is the engine's *standalone* context, so prefer **`liberate_preview`** (one call: creates the Studio site, imports the WXR + media, runs WP-default cleanup, activates the theme). Two Studio specifics the engine can't know: (1) `liberate_import` needs credentials — create an application password with `studio wp user application-password create` and pass it; (2) never point `studio wp import` / `wp eval-file` at the host `_liberations` path — Studio's PHP sandbox only reads inside the site (`/wordpress/...`), so let `liberate_preview` / `liberate_import` do the import rather than hand-rolling it.
- **Don't skip the final QA** — it's nested a level deeper and easy to stop short of after a long run. The chosen sub-skill ends with a mandatory parity step: **blocks** → `Read` `engineDir/skills/design-qa/SKILL.md` and complete its loop; **theme** → `replicate-theme`'s `liberate_compare` step. Run it fully; never substitute eyeballed screenshots, and honor the engine's honesty rule — no "looks good / matches" without measured parity.
- **`liberate_blockify_wxr`** — if it reports "no platform recorded", re-call with the platform from `liberate_detect` (e.g. `{ platform: "wix" }`); a no-op on platforms without a block recipe is expected, not a failure.
- **Timeouts (`MCP error -32001`) mean "still working," not "failed."** Any engine op can return `-32001` on the client side while it keeps running **inside the engine** — never treat it as a failure and never blindly re-invoke (a re-invoke often errors, e.g. `extraction already in progress`). For extraction/reconstruct ops (`liberate_extract`, `liberate_screenshot`, reconstruct), poll `liberate_status` with `{ outputDir }` until `running` is `false`, then continue from the output artifacts. For any other op that times out, wait and re-check its output/state before concluding it failed.

## Reporting

When the engine finishes, summarize its run report: the Studio site built into, the reconstruct path taken, counts (pages/posts/products/media), the parity/verdict, and anything flagged for manual attention.
168 changes: 168 additions & 0 deletions apps/cli/ai/tools/data-liberation.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,168 @@
import { existsSync } from 'fs';
import path from 'path';
import { Client } from '@modelcontextprotocol/sdk/client/index.js';
import { StdioClientTransport } from '@modelcontextprotocol/sdk/client/stdio.js';
import { Type } from 'typebox';
import { ensurePlaywrightChromiumInstalled } from '../browser-utils';
import { defineTool } from './define-tool';
import { textResult } from './utils';

const engineDir = path.join( import.meta.dirname, 'data-liberation-agent' );

let chromiumPromise: Promise< void > | null = null;

function ensureEngineChromium(): Promise< void > {
if ( ! chromiumPromise ) {
chromiumPromise = ( async () => {
const { chromium } = await import( 'playwright' );
const problem = await ensurePlaywrightChromiumInstalled( chromium );
if ( problem ) {
chromiumPromise = null; // allow a retry on the next tool call
console.error(
`[data_liberation] ${ problem } Browser-dependent steps (extract/screenshot/reconstruct) may fail.`
);
}
} )();
}
return chromiumPromise;
}

let clientPromise: Promise< Client > | null = null;

function getClient(): Promise< Client > {
if ( ! clientPromise ) {
clientPromise = connectClient().catch( ( error ) => {
clientPromise = null;
throw error;
} );
}
return clientPromise;
}

async function connectClient(): Promise< Client > {
if ( ! existsSync( path.join( engineDir, 'dist', 'mcp-server.js' ) ) ) {
throw new Error(
'Data Liberation engine is not compiled. Run `npm run cli:build` — it builds the ' +
'`data-liberation` workspace and bundles it into `dist/cli`.'
);
}

const transport = new StdioClientTransport( {
command: process.execPath,
args: [ path.join( engineDir, 'dist', 'mcp-server.js' ) ],
cwd: engineDir,
stderr: 'pipe',
} );
const client = new Client( { name: 'studio-code', version: '1.0.0' }, { capabilities: {} } );
await client.connect( transport );
return client;
}

interface DataLiberationResultContent {
type: string;
text?: string;
[ key: string ]: unknown;
}

// The model sometimes sends `args` as a JSON-encoded STRING instead of an object;
// forwarding that as MCP `arguments` fails the SDK schema ("expected record").
// Coerce nullish → {}, a JSON string → its parsed object, and reject anything else.
function normalizeArgs( raw: unknown ): Record< string, unknown > {
let value = raw;
if ( typeof value === 'string' ) {
const trimmed = value.trim();
if ( ! trimmed ) {
return {};
}
try {
value = JSON.parse( trimmed );
} catch {
throw new Error(
`data_liberation: \`args\` must be a JSON object, not a stringified JSON. Received: ${ trimmed.slice(
0,
200
) }`
);
}
}
if ( value === undefined || value === null ) {
return {};
}
if ( typeof value !== 'object' || Array.isArray( value ) ) {
throw new Error( 'data_liberation: `args` must be a JSON object.' );
}
return value as Record< string, unknown >;
}

export const dataLiberationTool = defineTool(
'data_liberation',
'Bridge to the Data Liberation engine, which extracts content from closed web platforms ' +
'(GoDaddy, Hostinger, HubSpot, Shopify, Squarespace, Webflow, Weebly, Wix) ' +
'and reconstructs it into a WordPress site. This tool forwards a single call to the engine; ' +
"the `/liberate` skill orchestrates the full sequence. Pass `tool: 'setup'` to get the paths " +
'to its skill files (the engine ships prebuilt with Studio).',
{
tool: Type.Optional(
Type.String( {
description:
"Engine MCP tool name, e.g. 'liberate_detect', 'liberate_discover', 'liberate_extract', " +
"'liberate_reconstruct_pages', 'liberate_install_theme'. Pass 'setup' to get the engine's " +
"skill-file paths, or 'list' to fetch the full catalog of engine tools with " +
'their argument schemas — consult it whenever you are unsure of a tool name or its arguments.',
} )
),
args: Type.Optional(
Type.Unknown( {
description:
'Arguments forwarded to the engine tool. MUST be a JSON OBJECT, not a stringified ' +
'JSON — e.g. { "url": "https://example.com" }, NOT "{\\"url\\":\\"…\\"}". ' +
"Site-targeting tools expect a Studio target, e.g. { kind: 'studio', sitePath: '/Users/you/Studio/my-site' }.",
} )
),
},
async ( args ) => {
if ( ! args.tool ) {
throw new Error(
'data_liberation: a `tool` is required. Pass "setup" to get the engine skill-file paths, ' +
'"list" for the tool catalog, or an engine tool name (e.g. "liberate_detect").'
);
}

if ( args.tool === 'setup' ) {
return textResult(
JSON.stringify( {
engineDir,
skillsDir: path.join( engineDir, 'skills' ),
liberateSkill: path.join( engineDir, 'skills', 'liberate', 'SKILL.md' ),
} )
);
}

const client = await getClient();

if ( args.tool === 'list' ) {
const listed = await client.listTools();
return textResult( JSON.stringify( listed.tools, null, 2 ) );
}

await ensureEngineChromium();

const result = await client.callTool( {
name: args.tool,
arguments: normalizeArgs( args.args ),
} );

const rawContent = Array.isArray( result.content )
? ( result.content as DataLiberationResultContent[] )
: [];
const text = rawContent
.map( ( part ) => ( part.type === 'text' ? part.text ?? '' : `[${ part.type }]` ) )
.join( '\n' );

if ( result.isError ) {
throw new Error( `Engine tool ${ args.tool } failed: ${ text }` );
}

return textResult( text );
}
);
2 changes: 2 additions & 0 deletions apps/cli/ai/tools/index.ts
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
import { emitChatArtifactWidgets } from 'cli/ai/chat-artifacts';
import { createPreviewTool } from './create-preview';
import { createSiteTool } from './create-site';
import { dataLiberationTool } from './data-liberation';
import { deletePreviewTool } from './delete-preview';
import { deleteSiteTool } from './delete-site';
import { exportSiteTool } from './export-site';
Expand Down Expand Up @@ -50,6 +51,7 @@ export const studioToolDefinitions: AnyStudioAgentTool[] = [
inspectDesignTool,
shareScreenshotTool,
installTaxonomyScriptsTool,
dataLiberationTool,
auditPerformanceTool,
auditSeoTool,
listConnectedRemoteSitesTool,
Expand Down
1 change: 1 addition & 0 deletions apps/cli/package.json
Original file line number Diff line number Diff line change
Expand Up @@ -44,6 +44,7 @@
"atomically": "^2.1.1",
"chokidar": "^5.0.0",
"cli-table3": "^0.6.5",
"data-liberation": "file:../../packages/data-liberation-agent",
"express": "^4.22.0",
"express-rate-limit": "^8.5.2",
"fs-extra": "^11.3.4",
Expand Down
32 changes: 31 additions & 1 deletion apps/cli/vite.config.base.ts
Original file line number Diff line number Diff line change
@@ -1,5 +1,6 @@
import { execSync } from 'child_process';
import { cpSync, existsSync, mkdirSync, writeFileSync } from 'fs';
import { resolve } from 'path';
import { relative, resolve, sep } from 'path';
import semver from 'semver';
import { defineConfig } from 'vite';
import packageJson from './package.json';
Expand Down Expand Up @@ -31,6 +32,16 @@ const phpSourceCodePath = resolve( __dirname, 'php' );
// The Skill tool loads skills from `<chunk dir>/skills` at runtime (see
// `ai/skills.ts`), so they must sit directly next to the built chunks.
const skillsSourcePath = resolve( __dirname, 'ai/skills' );

const dataLiberationSourcePath = resolve(
__dirname,
'..',
'..',
'packages',
'data-liberation-agent'
);
const repoRoot = resolve( __dirname, '..', '..' );

// The `studio ui` command serves the built browser UI (apps/ui `dist-local`)
// from `<chunk dir>/ui`, so it must sit next to the built chunks too. Built
// separately (`npm run build:local --workspace=apps/ui`); absent in API-only
Expand Down Expand Up @@ -61,6 +72,25 @@ export const baseConfig = defineConfig( {
if ( existsSync( skillsSourcePath ) ) {
cpSync( skillsSourcePath, resolve( outDir, 'skills' ), { recursive: true } );
}

execSync( 'npm -w data-liberation run build', { cwd: repoRoot, stdio: 'inherit' } );
cpSync( dataLiberationSourcePath, resolve( outDir, 'data-liberation-agent' ), {
recursive: true,
filter: ( src ) => {
const rel = relative( dataLiberationSourcePath, src );
if ( rel === '' ) {
return true;
}

const top = rel.split( sep )[ 0 ];
if ( top !== 'dist' && top !== 'package.json' && top !== 'skills' ) {
return false;
}
// Within dist/, drop build-only artifacts (types, tests, cache, maps).
return ! /\.(d\.ts|test\.js|js\.map|tsbuildinfo)$/.test( rel );
},
} );

if ( existsSync( localUiDistPath ) ) {
cpSync( localUiDistPath, resolve( outDir, 'ui' ), { recursive: true } );
}
Expand Down
2 changes: 2 additions & 0 deletions eslint.config.mjs
Original file line number Diff line number Diff line change
Expand Up @@ -94,6 +94,8 @@ export default defineConfig(
ignore: [
'@wp-playground/blueprints/blueprint-schema-validator',
'@modelcontextprotocol/sdk/server/stdio\\.js$',
'@modelcontextprotocol/sdk/client/index\\.js$',
'@modelcontextprotocol/sdk/client/stdio\\.js$',
],
},
],
Expand Down
1 change: 1 addition & 0 deletions package-lock.json

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

4 changes: 4 additions & 0 deletions packages/common/ai/slash-commands.ts
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,10 @@ export const AI_SKILL_COMMANDS: SkillSlashCommand[] = [
{ name: 'taxonomist', description: __( 'Optimize category taxonomy with AI' ) },
{ name: 'need-for-speed', description: __( 'Run a performance audit on a site' ) },
{ name: 'rank-me-up', description: __( 'Run an on-page SEO audit on a site' ) },
{
name: 'liberate',
description: __( 'Import & rebuild a site from a closed platform' ),
},
];

export function buildSkillInvocationPrompt( name: string ): string {
Expand Down
38 changes: 38 additions & 0 deletions packages/common/ai/tools.ts
Original file line number Diff line number Diff line change
Expand Up @@ -320,6 +320,7 @@ export function getToolDisplayName( name: string, input?: Record< string, unknow
need_for_speed: __( 'Audit performance' ),
rank_me_up: __( 'Audit SEO' ),
install_taxonomy_scripts: __( 'Install taxonomy scripts' ),
data_liberation: __( 'Data Liberation' ),
wpcom_request: __( 'WordPress.com API' ),
AskUserQuestion: __( 'Ask user' ),
Read: __( 'Read' ),
Expand All @@ -337,6 +338,41 @@ export function getToolDisplayName( name: string, input?: Record< string, unknow
}

const BASH_DETAIL_MAX_LENGTH = 60;
const DATA_LIBERATION_DETAIL_MAX_LENGTH = 80;

// The `data_liberation` tool forwards a single call to the Data Liberation
// engine's MCP server. Surface WHICH engine operation ran (`input.tool`, or
// `setup` when omitted) plus its arguments, so the tool-call line reads e.g.
// `Data Liberation liberate_detect {"url":"https://…"}` instead of a bare name.
// `args` is normally an object but the model sometimes sends stringified JSON,
// so accept both for display.
function getDataLiberationDetail( input: Record< string, unknown > ): string {
const tool = typeof input.tool === 'string' && input.tool ? input.tool : 'setup';

let argsObj: Record< string, unknown > | undefined;
const rawArgs = input.args;
if ( rawArgs && typeof rawArgs === 'object' && ! Array.isArray( rawArgs ) ) {
argsObj = rawArgs as Record< string, unknown >;
} else if ( typeof rawArgs === 'string' && rawArgs.trim() ) {
try {
const parsed: unknown = JSON.parse( rawArgs );
if ( parsed && typeof parsed === 'object' && ! Array.isArray( parsed ) ) {
argsObj = parsed as Record< string, unknown >;
}
} catch {
// Not JSON — fall through and show just the operation name.
}
}

if ( ! argsObj || Object.keys( argsObj ).length === 0 ) {
return tool;
}

const detail = `${ tool } ${ JSON.stringify( argsObj ) }`;
return detail.length > DATA_LIBERATION_DETAIL_MAX_LENGTH
? detail.slice( 0, DATA_LIBERATION_DETAIL_MAX_LENGTH - 1 ) + '…'
: detail;
}

function getAskUserDetail( input: Record< string, unknown > | undefined ): string {
const questions = input?.questions;
Expand Down Expand Up @@ -389,6 +425,8 @@ export function getToolDetail( name: string, input?: Record< string, unknown > )
const path = typeof input.path === 'string' ? input.path : '';
return [ method, path ].filter( Boolean ).join( ' ' );
}
case 'data_liberation':
return getDataLiberationDetail( input );
case 'wp_cli':
return typeof input.command === 'string' ? `wp ${ input.command }` : '';
case 'scaffold_theme':
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -15,9 +15,10 @@ const SRC = join(repoRoot, 'src');
const DIST = join(repoRoot, 'dist');

// Extensions of runtime (non-test, non-fixture) assets to mirror into dist/.
const RUNTIME_EXT = /\.php$/;
// Directories never copied (test fixtures / snapshots are not runtime).
const SKIP_DIR = /(^|\/)(__fixtures__|__snapshots__)(\/|$)/;
// .php: vendored WP helpers; .json: data assets read at runtime (e.g. core-block-attrs.json).
const RUNTIME_EXT = /\.(php|json)$/;
// Directories never copied (tests / fixtures / snapshots are not runtime).
const SKIP_DIR = /(^|\/)(__fixtures__|__snapshots__|__tests__)(\/|$)/;

let copied = 0;
function walk(dir) {
Expand Down
4 changes: 3 additions & 1 deletion packages/data-liberation-agent/tsconfig.json
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,9 @@
"skipLibCheck": true,
"declaration": true,
"jsx": "react-jsx",
"noEmitOnError": true
"noEmitOnError": true,
"incremental": true,
"tsBuildInfoFile": "dist/tsconfig.tsbuildinfo"
},
"include": ["src/**/*.ts", "src/**/*.tsx"],
"exclude": ["node_modules", "dist", "test", "scripts"]
Expand Down
Loading