Robots.txt troubleshooting: test the group, path and response
Write down the crawler and exact path before changing anything. A file that appears open can still select a different group or meet a server block.
1. The specific group does not inherit the wildcard restrictions
Google selects the most specific matching agent group, without combining it with *. Google’s group-selection rules explain why this draft allows Googlebot into the account path:
User-agent: *
Disallow: /account/
User-agent: Googlebot
Allow: /For this draft, Googlebot can request both /manuals/start and /account/profile. If the account restriction is intended for Googlebot, repeat it in that group:
User-agent: *
Disallow: /account/
User-agent: Googlebot
Allow: /
Disallow: /account/
Sitemap: https://example.com/sitemap.xmlTested result: /manuals/start remains allowed; /account/profile is blocked by the corrected draft. This is a crawl preference, not account security.
2. A longer Allow path can override a root block
User-agent: Googlebot
Disallow: /
Allow: /manuals/
Sitemap: https://example.com/sitemap.xml| Googlebot path | Tested result |
|---|---|
/manuals/start | Allowed |
/team/about | Blocked |
/Manuals/start | Blocked: path case differs |
Google uses the longest matching path; an Allow wins an equal-length conflict. Paths are case-sensitive. Source: Google rule precedence. Review exceptions against exact URLs rather than relying on an overall “blocked” label.
3. Keep Google Search and Google-Extended separate
User-agent: Googlebot
Allow: /
User-agent: Google-Extended
Disallow: /
Sitemap: https://example.com/sitemap.xmlGoogle-Extended controls certain Gemini training and grounding uses. It has no separate HTTP user-agent and does not control Google Search inclusion or rankings. Google’s crawler reference. This draft locally allows Googlebot at /manuals/start while blocking Google-Extended’s policy token.
For OpenAI, use the GPTBot / OAI-SearchBot / ChatGPT-User decision guide; do not apply Google’s product controls to a different provider.
4. A good draft is not proof of a working live file
- Fetch the exact origin’s
/robots.txt. It belongs at the host root and should be served as UTF-8 plain text. A subdomain needs its own file. Google’s file-location guidance. - If you receive HTML, a login screen or an unexpected status, fix the routing or CDN response before interpreting the draft. Save the URL, status and response body as separate evidence.
- Google generally treats robots.txt 4xx responses, except 429, as no file; 5xx or network failures can interrupt crawling. Do not use an error response as your blocking policy. Google’s status-code handling.
- If the file is correct but requests still fail, inspect server/CDN access rules and verified source IPs. A text checker cannot diagnose every network layer.
5. Do not use robots.txt as noindex or authentication
A disallowed public URL can still appear in Google results. Google must be able to fetch a page to read its noindex directive; use authentication for private content. Google’s limitations guide. Never put credentials or customer-specific URLs in a public crawler file.
The tests above cover exact groups and simple paths. They do not certify every robots.txt extension, CDN configuration or crawler implementation. Keep a dated policy diff and compare access logs after publishing.