Experimental incremental static builds#17084
Conversation
🦋 Changeset detectedLatest commit: 7785bea The changes in this PR will be included in the next version bump. This PR includes changesets to release 390 packages
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
github-actions
Bot
added
pkg: astro
![github-actions[bot]](https://cdn.statically.io/img/avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
matthewp
added
pr preview
matthewp
added
pr preview
github-actions
Bot
removed
the
pr preview
commit: |
|
@matthewp This is going to be a game changer for large sites! Can't wait to test the preview. Quick technical note on caching dist/ for Cloudflare: turning on Cloudflare's Native Build Cache (Settings > Build > Build cache > Enable) handles node_modules/.astro automatically, but my understanding is that it wipes the dist/ directory between builds. |
|
@deepanshu88 ok good to know. Which folders are cached isn't configurable? I considered caching the files in the node_modules/.astro/ folder but wanted to avoid duplicate writes. |
matthewp
added
the
pr preview
github-actions
Bot
removed
the
pr preview
|
@matthewp Unfortunately, Cloudflare's Native Build Cache isn't granularly configurable via settings or config files. For astro, it uses this cache directory - node_modules/.astro for reuse in subsequent builds. What if we only perform the duplicate write to node_modules/.astro/dist-cache/ when a specific CI-optimized flag or a fallback mode is triggered? Astro boots up and looks at ./dist. if It's completely empty, instead of re-render everything from scratch, Astro checks node_modules/.astro/project-cache/ and if it sees the backup HTML pages from Build1 are safe and sound. Before the actual compilation even begins, Astro instantly copy-pastes those backup HTML files out of node_modules/ and drops them right back into the empty ./dist folder. A 2x write penalty is a one-time infrastructure seeding cost. |
|
Yeah we'll probably need to do something like that. This PR is still early and being tested to make sure the gains are significant enough. Assuming it is we'll figure out the different CI environment stuff. |
|
Sounds like a good plan. Let me know if you need any large scale static sites to benchmark against once the PR is ready for public eyes - happy to help test it. |
|
@deepanshu88 It's ready to test now, with the caveat that you mentioned. So I would just test locally. Note that if you are on Astro 6 there are other changes in Astro 7 that improve perf, so you'd want to test your baseline, then do an Astro 7 upgrade, and then use this. |
matthewp
added
the
pr preview
github-actions
Bot
removed
the
pr preview
matthewp
added
the
pr preview
github-actions
Bot
removed
the
pr preview
|
@deepanshu88 The newest preview moves the cache into |
|
@matthewp That’s awesome to hear! Thanks for adding it and testing the same. Currently I am on Astro 6. I will upgrade to Astro 7 first to test the same. |
|
Took this PR for a real-site spin today on SetupThe site is pinned to Then I added Numbers (Windows host, single run each,
|
| Scenario | Wall | Built | Cached |
|---|---|---|---|
Cold (--force) |
904 s (~15 min) | 20,028 | 0 |
| Warm no‑op | 756 s | 13,881 | 6,147 |
Bump one package's version |
749 s | 13,884 (+3) | 6,144 (−3) |
The really nice finding 🥇
The content‑collection escape hatch (isContentDataIncrementalModule) works exactly as designed. Bumping the version field on one package JSON re‑rendered exactly 3 paths — the package landing, the type page, and the memberKind page for that one package — and left the other 6,144 cached entries untouched.
For contrast, on a smaller site of mine where I was loading data via plain import articles from './data/articles.json', the same kind of single‑entry edit invalidated all 240 article pages, because raw imports aren't marked as content‑data and get hashed into the per‑route dependencyHash. Bottom line: the content‑collection hatch is what makes this PR sing on real Starlight‑shaped sites. Worth flagging in the docs so folks lean into getCollection()-backed pipelines.
Caching profile (no‑op build, by route shape)
Where it caches (~6,147 pages, ~100 % of opted‑in routes):
| Route | Cached |
|---|---|
csharp/[package]/[type]/[kind]/index.html |
2,339 |
csharp/[package]/[type]/index.html |
1,509 |
csharp/[package]/index.html |
158 |
typescript/[module]/... |
2,115 |
samples/... |
26 |
Where it can't (yet) — a couple of opportunity areas worth chewing on:
-
Starlight markdown / MDX pages (~13,500 rebuilt). All the English docs, every localized mirror, integration pages, blog — they don't go through
getStaticPaths()so there's nocacheKeysurface to use. This is by far the biggest delta between "what I cache" and "what's in the build." -
Companion outputs from
astro:build:donehooks (~3,848 rebuilt). Specifically the.mdfilesstarlight-llms-txtemits next to each HTML page. The HTML version of these routes does cache, butrecordPathtracks the route's primary output, so the.mdcompanion re‑emits each build. The shape of the data is striking — the cached/built counts shift by exactly one layer at each level:Route layer Built ( .md)Cached ( .html)csharp/[package]/1,509 158 csharp/[package]/[type]/2,339 1,509 csharp/[package]/[type]/[kind]0 2,339
Both feel like additive features rather than redesigns — happy to dig in on either if it'd help.
Net for aspire.dev
- ~16 % wall‑clock saved on a true no‑op today (around 2.5 min per CI build). That's already a real win on a build this size, and the floor before any of the above gets addressed.
- Surgical invalidation works: a single‑package data change costs only the 3 pages that actually changed. 🎯
- Cherry‑picking the PR cleanly onto astro@6.x is straightforward, which made testing on a real Starlight site possible — worth keeping in mind for anyone else who wants to try this against ecosystem code that hasn't moved to 7 yet.
Really impressed with the design — cacheKey is a clean opt‑in surface, the content‑data exclusion is exactly the right hatch, and the persisted node_modules/.astro/dist/ cache makes this CI‑friendly out of the box. Excited to see this land. 🚀
If it'd be useful, I'm happy to:
- Open an issue with this benchmark + the
starlight-llms-txtcompanion‑output finding for a follow‑up. - Help spec a
cacheKeyanalogue for content collection / Starlight markdown routes (or test a prototype). - Run the same harness on other large sites to widen the signal.
cc @matthewp — thanks for picking this back up and shipping such a focused take on the design.
| * return posts.map(post => ({ | ||
| * params: { slug: post.slug }, | ||
| * props: { post }, | ||
| * cacheKey: post.updatedAt, |
There was a problem hiding this comment.
Why not show the newly added digest herre?
| * cacheKey: post.updatedAt, | |
| * cacheKey: post.digest, |
|
@IEvangelist Glad to hear that you have seen improvements! Your site is one of the ones I was thinking about when designing this. We definitely need to make it so that those 13k Starlight pages also get cached. Actually I'm pretty sure that Starlight does use |
3 participants
Changes
experimental.incrementalBuildflag that skips regenerating static pages whose module dependencies and user-providedcacheKeyare unchanged from the previous build.getStaticPaths()that include acacheKey. Static pages without acacheKeyare always re-rendered.dist/is preserved between builds instead of being emptied. Orphaned output files (pages removed fromgetStaticPaths) are cleaned up automatically.incremental-build.json) is stored innode_modules/.astro/alongside the existing content layer cache. Dependency hashes are computed from the Rolldown module graph including source file contents, so layout/component changes correctly invalidate cached pages.node_modules/.astro/anddist/must be preserved between builds for skipping to take effect. Ifdist/is missing, cached pages are re-rendered.Testing
incremental-build.test.tswith 9 assertions across 6 scenarios: first build, unchanged rebuild (cache hit), missing output file (forces re-render), changed dependency (invalidates all route paths), changedcacheKey(selective re-render), and flag-disabled regression (verifiesdist/is still emptied and no cache manifest is written).Docs
experimental.incrementalBuild.