Free robots.txt checker for AI crawlers

Check whether your robots.txt blocks the crawlers you actually want.

Paste a robots.txt file and get a plain-English audit for Googlebot, OAI-SearchBot, GPTBot, Google-Extended, Applebot, PerplexityBot, and CCBot. Useful before you publish crawler policy changes or debug AI-search visibility.

Check robots.txt Generate a cleaner file
Find mistakes Spot broad blocks that accidentally shut off search crawlers.
Separate use cases Review search, training-use, answer-engine, and dataset crawlers separately.
Copy the report Use the audit summary in launch notes or support threads.

AI crawler robots.txt checker

No login
Audit report 

          

        

      


      

        
Crawler access summary

        
This checker uses a practical robots.txt parser for common policy mistakes. Always test final rules with your hosting setup and official crawler docs before publishing.

        

          

            Status
            
Waiting

            
Run the checker to review crawler access.

          

        

        

          

            
              

                Crawler
                Use case
                Status
                Why it matters
              

            
            
              
Run the checker to see results.

            
          

        

      


      

        
Recommendations

        

          

            
Keep private content private

            
robots.txt is not security. Use authentication for customer files, staging, admin pages, and paid content.

          

          

            
Track before and after

            
Record the date you publish crawler-policy changes so traffic shifts can be explained later.

          

        

      


      

        
Useful references