Free tool
AI crawler checker: can GPTBot, ClaudeBot and PerplexityBot read your site?
Checks robots.txt and makes a live request to test whether ChatGPT, Claude and Perplexity can access and cite your page. Also separates AI training crawlers (GPTBot, ClaudeBot) from AI answer/search crawlers (OAI-SearchBot, Claude-SearchBot, PerplexityBot) — blocking one has no effect on the other.
What it does
- Parses robots.txt to spec — Group precedence, longest-match wins, wildcards and $ anchors. A named GPTBot group beats a blanket User-agent: * block — most free checkers get that backwards.
- Separates answers from training — Thirteen crawlers, split by what blocking each one actually costs you.
- Makes a real request — Three live fetches — a browser, ChatGPT Search and Anthropic — so a firewall block that robots.txt never mentions still shows up.
- Checks the exact path you gave it — Rules are per-path. /pricing can be blocked while the homepage is fine.
Questions
Does blocking GPTBot remove me from ChatGPT?
No. GPTBot collects training data. ChatGPT Search uses OAI-SearchBot to build its index and ChatGPT-User to fetch pages live when someone asks. You can block GPTBot and stay fully visible in ChatGPT — those are separate decisions, made with separate rules.
What is Google-Extended?
A control token, not a crawler. It opts your content out of Gemini model training and has no effect on Google Search or AI Overviews, which are served from the normal Googlebot index. Blocking Googlebot is what removes you from those.
Why does this tool make live requests as well as reading robots.txt?
Because they disagree. Cloudflare’s AI-scraper blocking, a WAF rule or a bot-management product will refuse an AI user-agent at the edge while robots.txt still says everything is allowed. Reading the file alone would report those sites as fine when no AI crawler can reach them.
Does a robots.txt rule actually stop a crawler?
It is voluntary. The major crawlers from OpenAI, Anthropic, Google and Perplexity honour it. Others are widely reported not to — Bytespider in particular. If you need enforcement rather than a request, that has to happen at your firewall.
Is this the same as the FAQLens scan?
No, and it is a much smaller question. This checks whether AI crawlers are permitted to read the page. The scan checks whether the page, once read, actually answers what a buyer needs to know before purchasing.
Readable is not the same as useful.
A crawler that can reach your pricing page still has to find an answer on it. FAQLens tests every buyer-critical question and tells you which ones your page leaves unanswered.