[codex] publish robots.txt

[basit3407]

What changed

• add static/robots.txt so the site publishes /robots.txt from the site root
• define explicit crawl rules with a global User-agent group
• block low-value or private routes from crawling: /search and /request-scopes
• advertise the public sitemap at https://api-docs.quran.foundation/sitemap.xml

Why

The site did not publish /robots.txt, so discoverability checks failed and crawlers had no explicit crawl policy for the docs site.

Impact

• /robots.txt will exist at the site root after deploy
• crawlers get a clear allow/disallow policy
• the docs sitemap becomes discoverable from the robots file

Root cause

The site root was missing a robots.txt asset.

Validation

• served static/robots.txt locally and verified GET /robots.txt returns 200
• verified the response Content-Type is text/plain

[Copilot]

Pull request overview

Adds a robots.txt static asset so the Docusaurus site publishes an explicit crawler policy at /robots.txt, including disallow rules for selected routes and a sitemap reference.

Changes:

• Add static/robots.txt served from the site root.
• Define User-agent: * rules allowing / while disallowing /search and /request-scopes.
• Advertise the sitemap at https://api-docs.quran.foundation/sitemap.xml.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[cloudflare-workers-and-pages]

Deploying qf-api-docs with    Cloudflare Pages

Latest commit:	b0a3a89
Status:	✅  Deploy successful!
Preview URL:	https://5d6dd0f3.qf-api-docs.pages.dev
Branch Preview URL:	https://codex-publish-robots-txt.qf-api-docs.pages.dev

View logs