Skip to content

[codex] add content signals to robots.txt#134

Merged
basit3407 merged 1 commit intomainfrom
codex/add-content-signals-robots-txt
Apr 20, 2026
Merged

[codex] add content signals to robots.txt#134
basit3407 merged 1 commit intomainfrom
codex/add-content-signals-robots-txt

Conversation

@basit3407
Copy link
Copy Markdown
Collaborator

@basit3407 basit3407 commented Apr 20, 2026

Summary

  • add a site-wide Content-Signal directive to robots.txt
  • keep the existing crawl rules and sitemap entry unchanged

Why

  • declare AI content usage preferences for training, search, and AI input
  • address the missing Content Signals directive reported for the site

Impact

  • crawlers that support Content Signals can see that the docs allow search and AI input, but disallow AI training
  • existing Allow, Disallow, and Sitemap behavior is unchanged

Validation

  • inspected git diff -- static/robots.txt
  • ran git diff --check
@basit3407 basit3407 marked this pull request as ready for review April 20, 2026 12:06
@cloudflare-workers-and-pages
Copy link
Copy Markdown

cloudflare-workers-and-pages Bot commented Apr 20, 2026

Deploying qf-api-docs with  Cloudflare Pages  Cloudflare Pages

Latest commit: a7c0c92
Status: ✅  Deploy successful!
Preview URL: https://61dfeb04.qf-api-docs.pages.dev
Branch Preview URL: https://codex-add-content-signals-ro.qf-api-docs.pages.dev

View logs
@basit3407 basit3407 merged commit c8ac4ff into main Apr 20, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

1 participant

@basit3407