Primary related guide or tool
Continue the workflow with this related LLMs.txt Kit resource.
/guides/gptbot-vs-oai-searchbot.htmlTraining opt-out policy page checklist
How to create a clear public policy page for AI training-use preferences without confusing it with search crawling.
Fast answer
If your goal is to communicate training-use preferences clearly, start with this framing: training-use policy, search crawling, and user-triggered fetching are often blended into one unclear statement. The useful deliverable is a public policy page outline.
This page is intentionally conservative. It treats crawler files, URL inspection, feeds, and server logs as discovery and measurement aids, not as guaranteed ranking levers.
When to use this playbook
Use it when publishers and companies with AI-use policies need a concrete next step and a page that can be linked from a hub, a community answer, a README, or a launch checklist. The page should help someone make a decision even if they never buy anything or contact the site owner.
The strongest pages in this topic cluster have three traits: they answer one narrow question, they include a copyable artifact, and they link to the relevant tool or proof page so the reader can act immediately.
Recommended workflow
- Separate search discovery from training-use controls.
- Name the user agents covered.
- Link the live robots.txt file.
- Record the date and owner of the policy.
Pre-publish checklist
- search policy clear.
- training policy clear.
- crawler tokens named.
- robots.txt linked.
Copyable working note
Use this as a starting point in a ticket, README, client note, or launch log. Edit it to match the real site before publishing.
We allow search crawlers for discovery.
We restrict training-use crawlers where appropriate.
Last reviewed: date.What not to count as proof
Do not count this setup as traffic by itself. A submitted sitemap, an IndexNow receipt, a crawler log hit, or an indexing request can show discovery work, but none of them proves rankings, impressions, clicks, conversions, or AI citations. Organic proof should come from Search Console, analytics, qualified referral evidence, or server logs interpreted for the right purpose.
The main pitfall for this topic is: Blocking search crawlers while trying to express a training-only preference.
Related resources
Next supporting resource
Continue the workflow with this related LLMs.txt Kit resource.
/guides/google-extended-vs-googlebot.htmlAll free tools
Continue the workflow with this related LLMs.txt Kit resource.
/tools/Proof dashboard
Continue the workflow with this related LLMs.txt Kit resource.
/proof.html