Primary related guide or tool
Continue the workflow with this related LLMs.txt Kit resource.
/guides/google-extended-vs-googlebot.htmlGoogle-Extended vs Googlebot: do not block the wrong crawler
How to separate Google Search crawling from Google-Extended controls when writing an AI-era robots.txt policy.
Fast answer
If your goal is to avoid blocking Google Search while making a deliberate AI-use decision, start with this framing: some teams block Googlebot while trying to opt out of unrelated AI uses. The useful deliverable is a short robots.txt review checklist for Googlebot and Google-Extended.
This page is intentionally conservative. It treats crawler files, URL inspection, feeds, and server logs as discovery and measurement aids, not as guaranteed ranking levers.
When to use this playbook
Use it when publishers and businesses that need organic search traffic but want a clear AI-use position need a concrete next step and a page that can be linked from a hub, a community answer, a README, or a launch checklist. The page should help someone make a decision even if they never buy anything or contact the site owner.
The strongest pages in this topic cluster have three traits: they answer one narrow question, they include a copyable artifact, and they link to the relevant tool or proof page so the reader can act immediately.
Recommended workflow
- Keep Googlebot rules aligned with search visibility goals.
- Document Google-Extended separately.
- Inspect the homepage in Search Console after changes.
- Watch logs for 403, 404, and blocked robots responses.
Pre-publish checklist
- Googlebot is not accidentally disallowed.
- Google-Extended policy is explicit.
- sitemap is reachable.
- Search Console can fetch key pages.
Copyable working note
Use this as a starting point in a ticket, README, client note, or launch log. Edit it to match the real site before publishing.
User-agent: Googlebot
Allow: /
User-agent: Google-Extended
Disallow: /What not to count as proof
Do not count this setup as traffic by itself. A submitted sitemap, an IndexNow receipt, a crawler log hit, or an indexing request can show discovery work, but none of them proves rankings, impressions, clicks, conversions, or AI citations. Organic proof should come from Search Console, analytics, qualified referral evidence, or server logs interpreted for the right purpose.
The main pitfall for this topic is: Using one broad Disallow rule when the real policy needs separate crawl intents.
Related resources
Next supporting resource
Continue the workflow with this related LLMs.txt Kit resource.
/tools/ai-crawler-robots-txt-checker.htmlAll free tools
Continue the workflow with this related LLMs.txt Kit resource.
/tools/Proof dashboard
Continue the workflow with this related LLMs.txt Kit resource.
/proof.html