Home / Guides

AI crawler log analysis: prove bots can reach the pages that matter

Search and AI crawler visibility should be proven with logs, not vibes. A small site can start with a simple access-log review: which bots arrived, which URLs they requested, and whether they received useful HTTP responses.

Proof levels

What to count

• Hits to /robots.txt, /sitemap.xml, and /llms.txt.
• Hits to primary landing pages, guides, tools, and templates.
• User agents for Googlebot, OAI-SearchBot, GPTBot, Applebot, PerplexityBot, and CCBot.
• Status codes: 200, 301/302, 403, 404, 429, and 5xx.
• Referrers and UTM parameters for legitimate community distribution.

Simple log review table

Metric                         Why it matters
/robots.txt hits               Crawlers checked access policy
/sitemap.xml hits              Crawlers discovered canonical URLs
/llms.txt hits                 AI-aware agents or humans checked context
200 status on core pages        Public pages are reachable
403/404 on core pages           Crawl or deployment problem
UTM campaign visits             Real distribution proof
Generator/copy events           Activation proof

Quick extraction commands

Use these locally on your own access logs. Redact sensitive values before sharing outputs.

# Find crawler-looking requests in an nginx or Apache combined log.
rg -i "Googlebot|OAI-SearchBot|GPTBot|Applebot|PerplexityBot|CCBot" access.log

# Count hits to discovery files.
rg "GET /(robots.txt|sitemap.xml|llms.txt)" access.log

# Look for crawler errors on important URLs.
rg -i "Googlebot|OAI-SearchBot|GPTBot|PerplexityBot|CCBot" access.log | rg " 4[0-9]{2} | 5[0-9]{2} "

# Summarize crawler hits by token when logs are small enough for shell review.
rg -io "Googlebot|OAI-SearchBot|GPTBot|Applebot|PerplexityBot|CCBot" access.log | sort | uniq -c | sort -nr

Bot verification notes

How this site proves it

LLMs.txt Kit writes server-side access events and funnel events on the VPS preview. The public proof file summarizes traffic sources, crawler classifications, campaign visits, and activation events. That makes the funnel auditable before the final domain is live.

Current proof files:

• Live proof JSON
• Distribution proof JSON
• Human-readable proof dashboard

7-day launch proof workflow

1. Day 0: confirm DNS, HTTPS, /robots.txt, /sitemap.xml, /llms.txt, and core pages return HTTP 200.
2. Day 0: submit sitemap in Search Console and IndexNow after the final domain is live.
3. Day 1: review logs for Googlebot, OAI-SearchBot, GPTBot, Applebot, PerplexityBot, and CCBot.
4. Day 2: check whether crawlers reach important guides, tools, templates, data pages, and proof files.
5. Day 3: compare crawler events with UTM-coded community distribution visits.
6. Day 5: import Search Console impressions and clicks if available.
7. Day 7: publish a short proof report showing crawler reach, referral traffic, tool activations, and unresolved issues.

Privacy checklist before sharing logs

• Remove or hash visitor IP addresses unless you need them for internal verification.
• Remove cookies, auth headers, session IDs, email addresses, and customer IDs.
• Trim query strings that might contain tokens, search terms, or private customer data.
• Aggregate counts when public proof does not need raw line-level evidence.

What not to count as success

• Self-refreshes that inflate pageviews.
• Automated search or click loops.
• Bot hits that never reach a useful page.
• Spam comments that create no real user engagement.

Sources

• Google request verification guide
• Google robots.txt limitations
• OpenAI crawler documentation
• Perplexity crawler documentation
• LLMs.txt Kit log analyzer
• AI crawler policy benchmark