Free tool · AI search
AI Crawler & Robots.txt Checker
See whether ChatGPT, Claude, Perplexity, Google and Bing are allowed to crawl a page: rule by rule from your robots.txt, plus a quick test for firewall or CDN blocks.
By Ajay Padhy · Updated · Free, no sign-up
Disclaimer
robots.txt is a request, not a lock: reputable crawlers follow it, others may not. The firewall test sends each bot's user-agent name only; real crawlers come from their own IP ranges, which your CDN may treat differently. Blocking Google-Extended affects Gemini training, not Google Search; blocking OAI-SearchBot can remove you from ChatGPT search results. Bot names and behaviour change, so check each provider's documentation before acting.
Privacy: URLs you check are fetched once to produce the result and aren't stored.
How to use the AI crawler checker
- Enter any page on your site, not just the homepage. robots.txt rules often differ by folder.
- The tool fetches your robots.txt, finds the rule group that applies to each crawler, and shows whether that exact path is allowed.
- It then requests the page with each crawler's user-agent to catch firewalls, CDN bot protection and rate limits that block bots before robots.txt even matters.
- Use the Generator tab to build a robots.txt that allows the crawlers you want and blocks the rest.
What the checker tests
| Check | What it tells you |
|---|---|
| robots.txt status | Whether the file exists. A 4xx means "no restrictions"; a 5xx can make Google pause crawling. |
| Rule match per crawler | Which group applies (the crawler's own group, or "*") and the exact Allow/Disallow line that decides it. |
| Firewall / CDN test | Whether requests with each bot's user-agent get a 403, 429 or a challenge page. |
| Page signals | Meta robots and X-Robots-Tag values, which control indexing separately from crawling. |
The matching follows the robots.txt standard (RFC 9309), the same way Google applies it: the most specific user-agent group wins, the longest matching rule wins, and on a tie Allow beats Disallow.
Search bots vs training bots: what blocking each one does
AI companies now run several crawlers with different jobs, and the difference matters:
- Search bots (Googlebot, Bingbot, OAI-SearchBot, Claude-SearchBot, PerplexityBot) decide whether you can appear and be cited in search and AI answers. Blocking them usually costs visibility.
- User-triggered fetchers (ChatGPT-User, Claude-User, Perplexity-User) load a page when someone asks an assistant about it.
- Training bots (GPTBot, ClaudeBot, CCBot and others) and the Google-Extended and Applebot-Extended controls decide whether your content can be used to train models. Blocking them doesn't remove you from search results.
Many sites block training while allowing search. That's a reasonable middle ground if you want AI visibility but not model training. My guide on how to rank in ChatGPT explains why OAI-SearchBot access matters. For the full process, including firewall and log checks, see how to check if ChatGPT, Claude and Perplexity can crawl your site.
Common robots.txt mistakes
- Blocking CSS and JavaScript that search engines need to render the page.
- A leftover "Disallow: /" from a staging site that went live.
- Blocking a folder in robots.txt and expecting it to disappear from Google. Disallow stops crawling, not indexing. Use noindex (and let the page be crawled) to remove it.
- Relying on robots.txt while a CDN blocks bots anyway. Some bot-protection settings block AI crawlers by default, whatever your robots.txt says. Check your CDN's bot or AI-crawler setting too.
- Typos in user-agent names. A group for "GPT-Bot" applies to nobody.
If a crawler is blocked by mistake
Fix the rule in robots.txt (or your CDN setting), then use the checker again. Google rereads robots.txt roughly once a day; other crawlers vary. If AI visibility matters to your business, the AI visibility audit covers crawler access alongside how AI assistants actually describe you.
FAQ
AI Crawler & Robots.txt Checker: common questions
Short answers to what people ask most about this tool and topic.
No. GPTBot is OpenAI's training crawler. Whether you appear in ChatGPT search results depends on OAI-SearchBot, which OpenAI controls separately. Blocking GPTBot while allowing OAI-SearchBot keeps you eligible for ChatGPT search while opting out of training.
No. Google-Extended isn't a separate crawler; it's a robots.txt token that controls whether content Google crawls can be used for Gemini training and grounding. Google Search crawling and ranking use Googlebot, which Google-Extended doesn't affect.
Your CDN, hosting firewall or security plugin may block requests that look like bots before robots.txt is considered. Check bot-protection or AI-crawler settings in your CDN. Note the test uses user-agent names only; real crawlers also come from known IP ranges, which some firewalls allow.
No. It's a widely respected convention. Reputable crawlers follow it, but it doesn't technically stop access. To truly restrict content, use authentication or firewall rules.
Google generally caches robots.txt for up to about a day. Other crawlers vary. Changes aren't instant, so allow some time before re-testing with live crawler logs.
Built by
SEO and AI search (GEO) consultant with 14+ years in digital growth. I build these tools for the checks I run on client sites every week. Need more than a quick check? See my GEO consulting or get in touch.
Found this useful? Link to it
If you mention the ai crawler & robots.txt checker in a guide, checklist or resource list, you can copy this link:
<a href="https://ajaypadhy.com/tools/ai-crawler-checker">AI Crawler & Robots.txt Checker</a>
More free tools
Have a growth problem worth solving?
Tell me where you're stuck. On a 30-minute intro call I'll tell you where I think the constraint is and what I'd look at first — whether or not we end up working together.