Skip to content

[PHP] Add zstd decoder stream for tar bundles - #3927

Merged
adamziel merged 3 commits into
trunkfrom
adamziel/tar-zst-decoder
Jul 7, 2026
Merged

[PHP] Add zstd decoder stream for tar bundles#3927
adamziel merged 3 commits into
trunkfrom
adamziel/tar-zst-decoder

Conversation

@adamziel

@adamziel adamziel commented Jul 7, 2026 •
edited
Loading

Copy link
Copy Markdown
Collaborator

What it does

Adds createDecodedTarStream(), a streaming codec layer that turns compressed TAR bundle bytes into a ReadableStream<Uint8Array> for StreamingTarParser from #3926.

This PR adds the zstddec fallback needed for future tar.zst WordPress core bundles. It does not switch any WordPress bundle URLs or boot paths to tar.zst yet.

Rationale

The parser from #3926 consumes plain TAR bytes. The bundle work in #3913 needs to feed it .tar.zst bytes, but browser support for native DecompressionStream('zstd') is not universal.

Keeping this as a small follow-up makes the dependency and decoder behavior reviewable on their own before the PR that changes the actual WordPress bundle format.

Implementation

createDecodedTarStream(compressed, codec):

  1. Accepts either a Uint8Array or ReadableStream<Uint8Array>.
  2. Uses native DecompressionStream when the runtime supports the requested codec.
  3. Falls back to zstddec/stream for codec === 'zstd'.
  4. Propagates decoder failures through the returned stream.

The zstd fallback uses ZSTDDecoder.decodeStreaming() to emit decoded TAR chunks without materializing the full decompressed archive in JS. Because zstddec consumes a synchronous iterable, ReadableStream input is first collected as compressed chunks before the decoder stream is returned.

Related: #3913

Testing instructions

npx nx test playground-wordpress --testFile=streaming-tar-extract.spec.ts
npx nx run playground-wordpress:lint
npx nx build playground-wordpress

The zstd fallback test uses a checked-in .tar.zst byte fixture generated with bsdtar and zstd, so it runs on Node 20 without node:zlib zstd compression support.
@adamziel
adamziel force-pushed the adamziel/tar-zst-parser branch from eb99759 to 9140ea5 Compare July 7, 2026 14:54
@adamziel
adamziel force-pushed the adamziel/tar-zst-decoder branch from cfaaf1e to 0b38e65 Compare July 7, 2026 14:56
@adamziel
adamziel force-pushed the adamziel/tar-zst-parser branch from 9140ea5 to 0b0f230 Compare July 7, 2026 16:16
@adamziel
adamziel force-pushed the adamziel/tar-zst-decoder branch from 0b38e65 to 938b848 Compare July 7, 2026 16:20
@adamziel
adamziel force-pushed the adamziel/tar-zst-parser branch from 0b0f230 to 09faba8 Compare July 7, 2026 18:36
@adamziel
adamziel force-pushed the adamziel/tar-zst-decoder branch from 938b848 to 69d9d07 Compare July 7, 2026 18:37
@adamziel
adamziel force-pushed the adamziel/tar-zst-parser branch from 09faba8 to 4ce5b5e Compare July 7, 2026 19:30
@adamziel
adamziel force-pushed the adamziel/tar-zst-decoder branch from 69d9d07 to 1ca7b06 Compare July 7, 2026 19:31
Base automatically changed from adamziel/tar-zst-parser to trunk July 7, 2026 19:46
@adamziel
adamziel force-pushed the adamziel/tar-zst-decoder branch from 1ca7b06 to 5155c1b Compare July 7, 2026 19:49
@adamziel
adamziel marked this pull request as ready for review July 7, 2026 19:52
@adamziel
adamziel requested review from a team, ashfame and Copilot July 7, 2026 19:52
Copilot AI reviewed Jul 7, 2026
View reviewed changes

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Note

Copilot was unable to run its full agentic suite in this review.

Adds a streaming decoder layer that converts compressed TAR bundle bytes into a decoded ReadableStream<Uint8Array> suitable for StreamingTarParser, including a zstddec streaming fallback for .tar.zst bundles where DecompressionStream('zstd') isn’t available.

Changes:

  • Introduces createDecodedTarStream() with native DecompressionStream first and zstddec/stream fallback for Zstandard.
  • Adds zstd round-trip tests (conditionally skipped if Node lacks zstd in node:zlib).
  • Adds zstddec dependency.

Reviewed changes

Copilot reviewed 3 out of 4 changed files in this pull request and generated 4 comments.

File Description
packages/playground/wordpress/src/streaming-tar-extract.ts Adds createDecodedTarStream() plus helpers to normalize inputs and decode tar bytes (native + zstd fallback).
packages/playground/wordpress/src/streaming-tar-extract.spec.ts Adds round-trip tests and helpers to validate zstd decoding into StreamingTarParser.
package.json Adds zstddec dependency needed for zstd fallback decoding.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment thread packages/playground/wordpress/src/streaming-tar-extract.ts Outdated
Comment thread packages/playground/wordpress/src/streaming-tar-extract.ts
Comment thread packages/playground/wordpress/src/streaming-tar-extract.spec.ts Outdated
Comment thread packages/playground/wordpress/src/streaming-tar-extract.spec.ts Outdated
@adamziel
adamziel merged commit 0a31d81 into trunk Jul 7, 2026
53 checks passed
@adamziel
adamziel deleted the adamziel/tar-zst-decoder branch July 7, 2026 20:39
@adamziel adamziel mentioned this pull request Jul 7, 2026
Merged
adamziel added a commit that referenced this pull request Jul 7, 2026
@adamziel
[PHP] Add streaming tar.zst WordPress bundle extraction (#3925)
21163fb 
## What it does

Adds the runtime extraction path for `tar.zst` WordPress core bundles
while keeping the existing `wordPressZip` / `unzipWordPress()` API
names.

This PR does not switch committed WordPress core bundle artifacts to
`tar.zst`. ZIP inputs from wordpress.org, custom URLs, trunk/nightly
downloads, plugins, themes, Blueprints, and GitHub artifacts stay on the
existing ZIP path.

## Rationale

#3926 added the TAR parser and #3927 added the zstd decoder stream. The
next step for #3913 is making WordPress boot able to consume a `tar.zst`
core bundle before changing the produced bundle artifacts.

Keeping runtime support separate from the artifact switch lets reviewers
validate the extraction behavior, API compatibility, and ZIP fallback
path independently.

## Implementation

`unzipWordPress()` now sniffs the first four bytes of the supplied
`File`:

1. zstd magic (`28 b5 2f fd`) routes through
`createDecodedTarStream(..., 'zstd')`.
2. Decoded TAR bytes stream into `extractTarStreamToPhp()` one entry at
a time.
3. ZIP and all other inputs continue through `unzipFile()` / PHP
`ZipArchive`.
4. `wordPressBundleFileCount` enables a tar.zst-only file-count parity
check when bundle metadata provides one.

The extractor writes files under `/tmp/unzipped-wordpress`, rejects
unsafe TAR entries through the parser, preserves `overwriteFiles=false`,
and handles `targetRoot='/'` as an absolute root instead of producing
relative paths.

## Testing instructions

```bash
npx nx test playground-wordpress --testFile=streaming-tar-extract.spec.ts
npx nx run playground-wordpress:lint
```

CI should run the broader package, typecheck, and browser/CLI coverage
for the rebased PR.
@adamziel adamziel mentioned this pull request Jul 7, 2026
Merged
adamziel added a commit that referenced this pull request Jul 7, 2026
@adamziel
[PHP] Replace WordPress core bundles with tar.zst (#3919)
d1ead35 
## What it does

Related to #3913.

Replaces the committed minified WordPress core bundle artifacts from
`wp-*.zip` to deterministic `wp-*.tar.zst` files.

This PR builds on the already-merged runtime/parser work: #3926, #3927,
and #3925. It does not change non-core ZIP flows. WordPress.org
downloads, custom URLs, `trunk`/`nightly`, plugins, themes, GitHub
artifacts, and Blueprint ZIP imports continue to use ZIP.

## Rationale

After #3916 removed the duplicated embedded `wordpress-static.zip`, the
remaining minified WordPress tree compresses better as a solid
zstd-compressed tar than as ZIP. The runtime can now stream-decode zstd
and parse TAR entries incrementally, so the committed core bundle can
switch formats without changing the public boot API.

Keeping remote and user-supplied archives as ZIP avoids expanding the
scope of this change beyond the committed Playground-managed core
bundles.

## Implementation

Adds a deterministic core-bundle build path that:

- re-containers the minified WordPress ZIP build output into
`wp-<version>.tar.zst`,
- writes `format`, `container`, `codec`, `size`, `sha256`, and
`fileCount` descriptors,
- verifies committed tar.zst artifacts against those descriptors,
- fails loudly for unsafe source archive paths instead of silently
dropping them,
- removes committed `wp-*.zip` artifacts for minified core versions, and
- teaches Vite/offline asset handling about `.tar.zst` and hashed `.zst`
files.

The generated descriptor still marks `trunk`/`nightly` as ZIP because
those versions are fetched directly from GitHub as ZIP archives instead
of using committed Playground core artifacts.

## Testing instructions

```bash
node packages/playground/wordpress-builds/build/build-tar-zst.mjs --verify
npx nx test playground-wordpress-builds
npx nx test playground-wordpress
npx nx test playground-remote
npx nx typecheck playground-remote
npx nx run-many -t lint -p playground-wordpress-builds,playground-remote,playground-wordpress
```
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment