Create the public files
Generate a concise llms.txt file and learn how to publish it safely without turning it into a keyword-stuffed doorway page.
Guides for llms.txt, AI crawler policy, and proof-based SEO
This hub groups every practical guide in the LLMs.txt Kit funnel. Start with the generator if you need an output today, then use the platform, crawler, and proof guides to make the site easier to crawl, understand, and measure.
Start here
Check crawler access
Understand how robots.txt controls Googlebot, OAI-SearchBot, GPTBot, Google-Extended, Applebot, PerplexityBot, and CCBot.
AI crawler robots.txt rulesTools and templates
All free tools
Browse the generator, checker, validator, log analyzer, sitemap converter, and policy badge tools from one crawlable hub.
Open the tools hubAI robots.txt generator
Create separate search, training-use, and broad crawler policies without blocking the wrong bot by accident.
Open the generatorAI crawler checker
Paste a robots.txt file and see whether important search and AI crawlers are allowed, blocked, or partially restricted.
Run the checkerAI crawler log analyzer
Paste access logs and prove whether Googlebot, OAI-SearchBot, GPTBot, Applebot, PerplexityBot, and CCBot reached key files.
Analyze crawler logsllms.txt validator
Paste a draft and check structure, public links, safety notes, concise scope, and obvious private-data risks.
Validate llms.txtSitemap to llms.txt
Paste sitemap XML or a URL list and generate a review-ready llms.txt draft from public canonical pages.
Convert sitemap to llms.txtAI crawler policy badge
Generate a nofollow trust badge for reviewed crawler policies without turning it into a link scheme.
Generate a badgellms.txt examples
Copy starter examples for SaaS, ecommerce, local service businesses, docs sites, and publishers.
Browse examplesAI crawler robots.txt examples
Copy policy templates for search visibility, training-use opt-outs, private docs, publishers, and staging sites.
Copy robots.txt examplesPlatform guides
Shopify
Pick collection, product, shipping, return, sizing, and policy pages for a clean ecommerce map.
llms.txt for ShopifyWordPress
Choose cornerstone posts, service pages, author pages, editorial policy, and contact pages instead of noisy archives.
llms.txt for WordPressSaaS docs
Map quickstarts, API references, changelogs, integrations, security pages, and versioned docs.
llms.txt for SaaS docsHelp centers
Map knowledge base categories, FAQs, troubleshooting hubs, status pages, and support escalation paths.
llms.txt for help centersFramer
Map Framer marketing pages, CMS collections, help content, case studies, policies, and safe publishing options.
llms.txt for FramerWebflow
Build a concise AI-readable map for marketing sites, case studies, pricing, help centers, and security pages.
llms.txt for WebflowWix
Map local service, booking, FAQ, review, and policy pages without exposing private member or cart URLs.
llms.txt for WixLocal business
Make service pages, NAP, hours, service areas, FAQs, and structured data easier to parse and measure.
AI search visibility for local businessConsultants and agencies
Package AI crawler readiness, llms.txt, robots.txt policy, Search Console, and log proof into a client-safe audit.
AI search visibility for consultantsCrawler policy and proof
GPTBot vs OAI-SearchBot
Separate OpenAI training-use crawling from ChatGPT search visibility decisions.
Compare OpenAI crawlersGoogle-Extended vs Googlebot
Understand the difference between Google Search crawling and Google AI-use controls.
Compare Google crawlersAI crawler benchmark
Use source-backed crawler policy notes and downloadable JSON/CSV data for OpenAI, Google, Apple, Perplexity, and Common Crawl.
Open the benchmarkUser-agent list
Scan AI crawler tokens by operator, category, robots.txt behavior, and practical policy note.
Open the user-agent listCopy-ready robots.txt examples
Use source-backed examples when you need to keep search crawlers open but opt out of selected AI data-use crawlers.
Open AI crawler robots.txt examplesAudit checklist
Verify robots.txt, sitemap.xml, llms.txt, indexability, internal links, and funnel tracking before launch.
Run the audit checklistLog analysis
Use server logs to prove crawlers and real users reached the pages and activation events that matter.
Analyze crawler logsFree traffic research
Use source-backed rules to separate legal free traffic tactics from fake search loops, doorway pages, and spammy link drops.
Read the traffic studyCommunity answer templates
Copy white-hat answers for relevant Reddit, GitHub, CMS, and webmaster discussions with direct tracked links.
Open answer templatesGitHub README snippets
Copy README, issue, PR, and docs snippets for projects that need llms.txt, crawler policy, and log-proof documentation.
Open GitHub snippetsAI tool directory submission kit
Copy safe descriptions and curator pitches for relevant AI tools, SEO tools, Product Hunt-style launches, newsletters, and directories.
Open directory kitDirectory target queue
Review verified and manual-review launch, AI tool directory, software directory, and developer community targets before final-domain distribution.
Open target queueNew domain SEO launch checklist
Use DNS, HTTPS, Search Console, sitemap, IndexNow, proof files, and first-distribution checks to move a new domain from preview to production.
Open launch checklistAI crawler policy changelog
Track source-backed policy checks for OAI-SearchBot, GPTBot, Googlebot, Google-Extended, Applebot, PerplexityBot, CCBot, llms.txt, and IndexNow.
Open the changelogMonthly updates
Keep crawler rules and AI-search advice fresh without changing dates on unchanged content.
Read the July update
Proof principle: every guide should be discoverable from this hub, listed in the sitemap, and included in the public live checks before final-domain Search Console submission.