[codex] enable markdown negotiation for agents

[basit3407]

What changed

• generate markdown siblings for built HTML pages during the docs build
• add Cloudflare Pages middleware to serve markdown when requests send Accept: text/markdown
• keep HTML as the default response for browser requests and preserve Vary: Accept
• declare the HTML parsing libraries as build-only dependencies
• add tests for markdown asset generation and response negotiation

Why

The site served HTML only, so agent clients requesting markdown could not negotiate a markdown representation of the docs. This change adds explicit markdown-for-agents handling without changing the default browser behavior.

Impact

Agents and other markdown-capable clients can request docs as markdown with Content-Type: text/markdown, while normal browser traffic continues to receive HTML.

Root cause

The docs build emitted only HTML and the Cloudflare runtime had no content negotiation path for markdown requests.

Validation

• node --test tests\\api-demo-panel-ui.test.cjs tests\\prelive-docs-config.test.cjs tests\\user-related-env-paths.test.cjs tests\\user-related-env-ui.test.cjs tests\\markdown-negotiation.test.cjs
• yarn install --frozen-lockfile --ignore-engines

[cloudflare-workers-and-pages]

Deploying qf-api-docs with    Cloudflare Pages

Latest commit:	3fd78df
Status:	✅  Deploy successful!
Preview URL:	https://cae1a68c.qf-api-docs.pages.dev
Branch Preview URL:	https://codex-markdown-for-agents.qf-api-docs.pages.dev

View logs

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 9eb41ad6af

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[Copilot]

Pull request overview

Adds first-class Markdown content negotiation for agent clients by generating .md siblings during the docs build and serving them via Cloudflare Pages middleware when requests include Accept: text/markdown.

Changes:

• Generate Markdown variants (.md) next to built HTML pages during the Docusaurus build.
• Add Cloudflare Pages middleware to negotiate Markdown responses while preserving Vary: Accept and defaulting to HTML.
• Add tests covering Accept parsing, path mapping, markdown generation, and runtime negotiation behavior.

Reviewed changes

Copilot reviewed 7 out of 7 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
tests/markdown-negotiation.test.cjs	Adds coverage for Accept parsing, markdown asset mapping, markdown generation, and runtime response negotiation.
src/markdown-negotiation-shared.cjs	Shared helpers/constants for Accept parsing, path mapping, and header manipulation.
src/markdown-negotiation-runtime.cjs	Implements the response negotiation logic used by Pages middleware.
src/build-markdown.cjs	Adds HTML→Markdown conversion plus export of .md siblings for built HTML pages.
plugins/llms-txt-plugin.js	Hooks markdown sibling generation into the postBuild step.
package.json	Adds HTML parsing libs needed for markdown generation.
functions/_middleware.js	Pages middleware that serves markdown when negotiated via Accept: text/markdown.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.