Primary related guide or tool
Continue the workflow with this related LLMs.txt Kit resource.
/guides/gptbot-vs-oai-searchbot.htmlGPTBot vs OAI-SearchBot in plain English
A simple policy guide for deciding whether to allow GPTBot, OAI-SearchBot, or both in robots.txt.
Fast answer
If your goal is to separate search discovery from model training policy, start with this framing: blocking every OpenAI user agent can remove eligibility for search surfaces that a business actually wants. The useful deliverable is a crawler policy note that treats search retrieval and training-use controls separately.
This page is intentionally conservative. It treats crawler files, URL inspection, feeds, and server logs as discovery and measurement aids, not as guaranteed ranking levers.
When to use this playbook
Use it when site owners who want ChatGPT search eligibility but also want a clear training-use policy need a concrete next step and a page that can be linked from a hub, a community answer, a README, or a launch checklist. The page should help someone make a decision even if they never buy anything or contact the site owner.
The strongest pages in this topic cluster have three traits: they answer one narrow question, they include a copyable artifact, and they link to the relevant tool or proof page so the reader can act immediately.
Recommended workflow
- List each OpenAI crawler token separately.
- Decide whether search discovery and training use have different risk profiles.
- Test robots.txt with a parser before publishing.
- Check access logs for real crawler visits after launch.
Pre-publish checklist
- OAI-SearchBot has a deliberate rule.
- GPTBot has a deliberate rule.
- ChatGPT-User is not confused with autonomous crawling.
- policy notes are documented for future maintainers.
Copyable working note
Use this as a starting point in a ticket, README, client note, or launch log. Edit it to match the real site before publishing.
User-agent: OAI-SearchBot
Allow: /
User-agent: GPTBot
Disallow: /private/What not to count as proof
Do not count this setup as traffic by itself. A submitted sitemap, an IndexNow receipt, a crawler log hit, or an indexing request can show discovery work, but none of them proves rankings, impressions, clicks, conversions, or AI citations. Organic proof should come from Search Console, analytics, qualified referral evidence, or server logs interpreted for the right purpose.
The main pitfall for this topic is: Copying a viral blocklist without checking which product surface each token represents.
Related resources
Next supporting resource
Continue the workflow with this related LLMs.txt Kit resource.
/tools/ai-crawler-robots-txt-checker.htmlAll free tools
Continue the workflow with this related LLMs.txt Kit resource.
/tools/Proof dashboard
Continue the workflow with this related LLMs.txt Kit resource.
/proof.html