Find out what User-agent: GPTBot means, compare OpenAI crawlers, and choose a robots.txt policy without confusing model-training preferences with ChatGPT search discovery.
GPTBot is OpenAI's crawler for content that may be used to train its generative AI foundation models. In a robots.txt file, User-agent: GPTBot starts a group of rules for that crawler. It is not the same as OAI-SearchBot, which is used for ChatGPT search discovery.
In robots.txt: use the token GPTBot, not the entire browser-style HTTP user-agent string.
In server logs: a request may contain GPTBot/ followed by a version number. The version can change; the string alone does not prove who sent it.
For separate choices: you can disallow GPTBot while allowing OAI-SearchBot. Neither setting guarantees inclusion, citations, or rankings.
OpenAI guidance checked on : Overview of OpenAI Crawlers. This review date applies to the OpenAI explanations on this page, not a fresh audit of every provider in the lookup.
Search crawler user agents
Enter a crawler token, a line such as User-agent: GPTBot, or an HTTP user-agent string. You can also search a provider or use case, such as OpenAI or user triggered. This is a text lookup, not a live IP identity check.
All crawler records are listed below. Enable JavaScript to filter them.
Lookup result and next step
AI crawler user-agent lookup ready.
Use /data/ai-crawler-user-agent-lookup-pack.json for machine-readable crawler records.
Use /.well-known/ai-crawler-user-agent-lookup-pack.json as the stable discovery path.
GPTBot vs OAI-SearchBot vs ChatGPT-User
Choose the control for the job you want to manage. A training-use preference is not a search opt-out, and a user-requested visit is not an automatic crawl.
OpenAI crawler roles and the limits of each control
Agent
Purpose
What to control
GPTBot
Automatic crawling of content that may be used for model training.
Use the GPTBot robots.txt group to express your crawling and training-use preference.
OAI-SearchBot
Discovery for ChatGPT search features.
Use the OAI-SearchBot group for automatic search crawling. OpenAI says opted-out sites may still appear as navigational links.
ChatGPT-User
Certain page visits requested by users of ChatGPT or Custom GPTs.
OpenAI says robots.txt rules may not apply to user-initiated actions. Use OAI-SearchBot, not ChatGPT-User, to manage search opt-outs.
This is a fragment, not a replacement for your robots.txt file. Merge it with existing groups and retain any restrictions you need. The OAI-SearchBot example allows the whole path space; do not use it unchanged if some paths should remain disallowed. Robots.txt does not protect private content or override login requirements or firewall blocks.
What if I want to allow GPTBot?
If your policy permits GPTBot crawling, use an appropriate GPTBot group such as the example below, while retaining any path-specific exclusions your site needs. Make the OAI-SearchBot decision separately.
User-agent: GPTBot
Allow: /
Before changing a live file, review the merged rules with the AI crawler robots.txt checker and test representative public and restricted paths. OpenAI says its search systems can take approximately 24 hours to adjust after a robots.txt update; that is not a promise of a crawl, citation, or ranking.
How do I verify a GPTBot request in my logs?
A GPTBot user-agent match is a clue, not identity proof: user-agent strings can be spoofed. This lookup identifies text; it does not contact OpenAI or validate the source IP of a request.
Find the request time, URL path, response status, user agent, and source IP in your own server or edge logs. Keep sensitive log details private.
If a CDN or reverse proxy sits in front of your server, use its trusted client-IP information. Do not trust an arbitrary client-supplied forwarding header.
Compare the request's source IP with OpenAI's current GPTBot IP ranges. OAI-SearchBot and ChatGPT-User have their own published lists. Check membership in the published network ranges, not just whether an address looks similar.
Check the request path and status before drawing conclusions. A visit to robots.txt is not a fetch of your article; a blocked request is not a successful content fetch; crawler traffic is not human traffic.
OpenAI notes that robots.txt requests may include an extra robots.txt marker in the HTTP user agent. Do not hard-code a single full user-agent version as proof of identity. For a next step, use the log analyzer on a sanitized sample, then verify identity against the provider's current guidance.
Common GPTBot questions
Does blocking GPTBot remove my website from ChatGPT search?
Not by itself. OpenAI documents GPTBot and OAI-SearchBot as separate controls. You can disallow GPTBot while allowing OAI-SearchBot. Search inclusion still is not guaranteed; check the OAI-SearchBot policy and server access separately.
Should I paste the full HTTP user-agent string into robots.txt?
No. The group starts with a product token, such as User-agent: GPTBot. Browser-style strings and version numbers are examples of what you might see in HTTP logs, not the token you need for this robots.txt group.
Does a GPTBot visit prove my content was used for training or cited?
No. A verified request can show that a crawler requested a particular resource. It does not prove a later training use, a citation in an answer, a human visit, or a search ranking improvement.
Will robots.txt stop every bot or protect private pages?
No. Robots.txt communicates rules to cooperating crawlers; it is not access control. Protect private content with authentication and appropriate server controls. OpenAI also states that robots.txt rules may not apply to user-initiated ChatGPT-User requests.
Automatic search crawler for ChatGPT search surfaces.
Default: Allow when ChatGPT search visibility matters; decide separately from GPTBot.
Verify: Match the OAI-SearchBot token, then verify against the published OpenAI searchbot IP JSON before using it as identity proof.
Copy-ready note
User-agent: OAI-SearchBot
Allow: /
Example HTTP user agent (version numbers may change): Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36; compatible; OAI-SearchBot/1.3; +https://openai.com/searchbot
Crawler for content that may be used to improve OpenAI generative AI foundation models.
Default: Decide from training-use policy; do not block OAI-SearchBot just because GPTBot is blocked.
Verify: Match GPTBot, then verify against OpenAI's published GPTBot IP JSON where identity matters.
Copy-ready note
User-agent: GPTBot
Disallow: /
Example HTTP user agent (version numbers may change): Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.3; +https://openai.com/gptbot
Example HTTP user agent (version numbers may change): Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; ChatGPT-User/1.0; +https://openai.com/bot
OpenAI ads landing-page safety and relevance validation.
Default: Only relevant if submitting ads on ChatGPT; do not confuse with organic search crawling.
Verify: Verify against the published OAI-AdsBot IP JSON when ad review traffic matters.
Copy-ready note
User-agent: OAI-AdsBot
Allow: /
Example HTTP user agent (version numbers may change): Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; OAI-AdsBot/1.0; +https://openai.com/adsbot
Default: Allow if Perplexity search/answer visibility matters, and whitelist published IP ranges if a WAF blocks it.
Verify: Match PerplexityBot and verify against Perplexity's published IP range JSON.
Copy-ready note
User-agent: PerplexityBot
Allow: /
Example HTTP user agent (version numbers may change): Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; PerplexityBot/1.0; +https://perplexity.ai/perplexitybot)
User-requested fetcher for Perplexity user actions.
Default: Monitor separately from PerplexityBot; verify IP ranges for WAF allow rules.
Verify: Treat as user-triggered and verify against Perplexity's published Perplexity-User IP JSON.
Copy-ready note
# Perplexity-User is user-triggered. Review WAF rules and logs separately.
Example HTTP user agent (version numbers may change): Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Perplexity-User/1.0; +https://perplexity.ai/perplexity-user)
For machine-readable records and integration examples, use the links below. These are companion datasets, not live verification of a request or a substitute for each provider's current documentation.