Frequently asked questions
What does a robots.txt checker do?
It fetches your site's /robots.txt and audits it: which crawlers are allowed or blocked, whether a wildcard Disallow: / blocks everything, whether a sitemap is declared, and whether any directives have syntax errors. Our checker goes further than most by testing the AI crawlers that matter for AI search visibility — GPTBot, ClaudeBot, PerplexityBot, Google-Extended, CCBot and Bytespider.
Why would AI crawlers be blocked from my site?
Usually by accident. A Disallow: / left over from staging, a CDN bot-management rule, or a copied robots.txt template can block GPTBot or ClaudeBot without anyone noticing. If an AI crawler cannot fetch your pages, it cannot cite them in answers — so a silent block directly costs you AI visibility.
Is blocking AI crawlers ever the right choice?
Sometimes. If you do not want your content used for model training, blocking training-specific bots like Google-Extended or CCBot is a legitimate choice. But blocking answer-citation bots (GPTBot, ClaudeBot, PerplexityBot) trades away AI search visibility. The key is to block deliberately, not accidentally — this checker shows you exactly where you stand.
What should a good robots.txt contain?
At minimum: explicit User-agent groups for the crawlers you care about, Disallow rules only for paths you truly want hidden (admin, login, staging), and a Sitemap: line pointing at your sitemap.xml. Keep it under a few KB, avoid unknown directives, and never use Disallow: / on a production site.