OpenAI
OAI-SearchBot is the token to review when ChatGPT search visibility matters. GPTBot is a separate policy decision for training-use crawling.
AI crawler policy changelog
This page is the recurring update hub for AI crawler rules. Use it to review which crawler tokens matter, what each control is meant to affect, and what to test before asking Google Search Console, Bing, or community users to discover a new site.
The useful traffic angle is simple: crawler policy changes are confusing, and site owners need a clean monthly checklist they can trust. That makes this page a source-backed asset, not a fake freshness trick.
June 2026 snapshot
No ranking shortcut is claimed here. The current safe interpretation is: keep Googlebot and other search crawlers open for pages you want discovered, separate search visibility from model-training controls, keep /llms.txt concise, and measure real crawler hits in server logs.
Watchlist
Googlebot affects Google Search and related features. Google-Extended is a control token for Gemini / Vertex AI usage and Google says it does not affect Search inclusion or ranking.
Google AI features
Google's guidance for generative AI search still points back to core SEO systems: crawlability, useful content, structured data that matches visible content, snippets, and page experience.
Apple
Applebot and Applebot-Extended should be treated as separate checks: discovery/use surfaces versus broader data-use controls.
Perplexity
PerplexityBot and Perplexity-User should be checked explicitly if answer-engine visibility or user-triggered retrieval matters.
Common Crawl
CCBot is an open dataset participation decision. It is useful to document, but it should not be confused with immediate search visibility.
Monthly checklist
Copy this checklist into a GitHub issue, client note, or launch runbook. The copy action is tracked as a funnel activation event so we can prove this page is doing more than attracting passive page views.
AI crawler policy monthly check 1. Fetch /robots.txt and confirm it returns HTTP 200. 2. Confirm Googlebot is allowed for public pages that should appear in Google Search. 3. Confirm OAI-SearchBot policy matches ChatGPT search visibility goals. 4. Confirm GPTBot policy matches model-training-use preferences. 5. Confirm Google-Extended policy is documented separately from Googlebot. 6. Confirm Applebot, Applebot-Extended, PerplexityBot, Perplexity-User, and CCBot rules are intentional. 7. Fetch /sitemap.xml and confirm newly published URLs are listed. 8. Fetch /llms.txt and confirm it only links to public, canonical, useful pages. 9. Run a server-log check for Googlebot, OAI-SearchBot, GPTBot, Applebot, PerplexityBot, and CCBot. 10. Record Search Console impressions, clicks, indexed pages, and any crawl errors.
What changed in this launch
Added live proof
The preview deployment exposes live proof JSON and a proof dashboard for HTTP checks, crawler-classified visits, and funnel events.
Added distribution proof
The launch pack exposes distribution proof with UTM-tracked community placements so real referral traffic can be separated from tests.
Added legal answer templates
The community answer templates page gives copyable answers that link to tools only after solving the user's question.
Added IndexNow packet
The key file and packet are ready, but submission should wait until the final domain, sitemap, and key file all return HTTP 200.
Use the tools
- Paste a robots.txt file into the AI crawler checker
- Generate an AI crawler robots.txt policy
- Validate an llms.txt draft
- Compare crawler tokens in the benchmark
FAQ
Should I block every AI crawler?
Not automatically. Separate search visibility, answer-engine retrieval, training-use crawling, and open dataset participation. Blocking every token is simple, but it can reduce discovery in places where you may want visibility.
Does this replace Search Console?
No. This changelog is a maintenance page. Search Console is still needed for verification, sitemap submission, indexing diagnostics, impressions, clicks, and query data.