GPTBot, OAI-SearchBot and ChatGPT-User: choose the right control
Start with the request’s purpose. A log entry containing “OpenAI” is not enough to decide which access policy to apply.
Three agents, three jobs
| Token | Purpose | Control to review |
|---|---|---|
GPTBot | Content that may be used for foundation-model training. | Its own robots.txt group. |
OAI-SearchBot | Discovery for ChatGPT search. | Its own group and network access. |
ChatGPT-User | Visits initiated by certain user actions. | Authentication and access policy; robots.txt may not apply. |
The search and training settings are independent. ChatGPT-User is not the search opt-out control. Opting out of OAI-SearchBot excludes search answers, although navigational links may remain. Source: OpenAI crawler documentation.
Worked policy: search allowed, training declined
This illustrative policy keeps public search access open and asks GPTBot not to crawl. Replace the example sitemap host. Protect accounts with authentication regardless of crawler rules.
User-agent: OAI-SearchBot
Allow: /
Disallow: /account/
User-agent: GPTBot
Disallow: /
Sitemap: https://example.com/sitemap.xml| Local rule test | Expected result |
|---|---|
OAI-SearchBot → /manuals/start | Allowed |
OAI-SearchBot → /account/profile | Blocked by this draft |
GPTBot → /manuals/start | Blocked by this draft |
These are tested local rule expectations, not observations of OpenAI traffic or evidence of citations. Other crawlers need their own policy review. Work through robots.txt group and path mistakes before combining this with existing rules.
Verify a request before trusting its name
User-agent versions can change. Compare the request’s source IP with the relevant published ranges; a token alone is not identity proof. OpenAI publishes the reference ranges.
- Record timestamp, requested path, source IP and response status from your server or CDN logs. Redact personal data before sharing.
- Identify the token with the lookup tool; then follow its provider source link.
- For OpenAI, use the corresponding searchbot, gptbot or chatgpt-user ranges. Keep unmatched traffic marked “unverified”.
- Compare the response with your intended policy. An allowed rule alongside a CDN 403 needs a network review, not another llms.txt file.
OpenAI says search changes can take about 24 hours after a robots.txt update. That is not a citation deadline. Allowing access does not guarantee selection or rankings. Source: OpenAI.
Keep access and search eligibility separate
robots.txt is not an authentication boundary. Use access controls for private files; blocking crawl alone does not reliably remove a public URL from Google results. Google explains these limitations.