How to Check if ChatGPT, Claude and Perplexity Can Crawl Your Site

If your site never shows up in ChatGPT, Claude or Perplexity answers, check access before you rewrite any content. A common cause isn't robots.txt at all: it's a firewall, CDN setting or security plugin quietly returning a 403 to bots it doesn't recognise, including the AI search crawlers.

People often check robots.txt, see nothing blocked, and stop. Here are the five checks I run instead, in order.

Know which bots matter

Each AI company runs several bots, and they do different jobs. Blocking the wrong one either costs you visibility or doesn't do what you intended.

What the providers say:

• OpenAI: sites that opt out of OAI-SearchBot "will not be shown in ChatGPT search answers". Disallowing GPTBot means your content shouldn't be used to train its models. ChatGPT-User acts on a user's request, so robots.txt rules "may not apply". (OpenAI bots documentation)
• Anthropic: its bots honour robots.txt and the Crawl-delay extension. Blocking Claude-SearchBot or Claude-User may reduce your visibility in Claude's search results, while blocking ClaudeBot opts future content out of training. (Anthropic crawler documentation)
• Perplexity: PerplexityBot surfaces and links sites in its search results and isn't used to train foundation models. Perplexity-User "generally ignores robots.txt rules" because a user requested the fetch. (Perplexity crawler documentation)

A common, reasonable setup is to allow the search bots and block the training bots. You stay visible in AI answers without contributing to model training.

Check 1: robots.txt rules for each bot

Open yourdomain.com/robots.txt and read it the way a crawler does:

1. A crawler looks for the group with the most specific User-agent match for its name. If there's a User-agent: GPTBot group, GPTBot follows that group and ignores User-agent: *.
2. Within its group, the most specific (longest) matching Allow or Disallow path wins.
3. Each subdomain needs its own robots.txt. blog.yourdomain.com doesn't inherit the rules from www.

The mistakes I find most often:

• A blanket User-agent: * / Disallow: / left over from a staging site.
• Copy-pasted "block all AI bots" lists that include OAI-SearchBot, Claude-SearchBot and PerplexityBot along with the training bots.
• A rule like Disallow: /*? that also blocks important filtered or paginated URLs.

A setup that allows AI search but blocks training looks like this:

User-agent: OAI-SearchBot
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: *
Allow: /

Sitemap: https://www.yourdomain.com/sitemap.xml

Reading rules by hand gets error-prone once a file has a dozen groups. My free AI crawler and robots.txt checker tests a URL against each bot and shows which rule decided the result.

Check 2: firewall, CDN and security plugins

This is the check most people skip. robots.txt is a request; a firewall is a lock. If your CDN, hosting firewall or WordPress security plugin blocks a bot, robots.txt never comes into play.

Things to look at:

• CDN bot settings. Cloudflare, for example, has AI-crawler blocking and bot-fight features that can be switched on at the account level. Check what's enabled, and allow the search bots you want.
• Security plugins and WAF rules that block "unknown bots" or rate-limit aggressively.
• Country or ASN blocks. AI crawlers run from cloud data centres, often in the US.

A quick first test is to request a page with each bot's user-agent name and compare the status code with a normal browser request. A 403, 429 or a challenge page for the bot means something upstream is blocking it. This only tests the name: real crawlers come from their providers' published IP ranges, which your CDN may treat differently, so confirm with your logs (check 4). The checker above runs this user-agent test for you.

Check 3: is your content in the raw HTML?

Most AI crawlers fetch the HTML your server sends and don't run JavaScript the way a browser does. If your site renders its main content in the browser (a typical React or Vue single-page app), the bot can reach the page and still see an empty shell.

Test it by viewing the page source (not the inspector) and searching for a sentence from your main content. If it isn't there, the content is client-side rendered. My CSR vs SSR checker counts the words, headings, links and structured data in the raw HTML and gives a verdict.

The fix is server-side rendering or pre-rendering for the pages you want cited. Frameworks such as Next.js and Nuxt do this by default.

Check 4: server logs

Logs show what actually happened, rather than what should happen. Filter your access logs for the bot names above and look at:

• Which bots visit at all. If OAI-SearchBot never appears, the cause may be a block upstream, or simply that it hasn't found you yet.
• Status codes. Lots of 403s or 429s for a bot means it's being blocked or rate-limited.
• Which URLs they request. If bots only ever hit your homepage, your internal linking and sitemap need work.

Remember that anyone can fake a user-agent. To confirm a visit is real, check the IP against the ranges each provider publishes (OpenAI, Anthropic and Perplexity all document theirs).

Check 5: page-level signals

Finally, check the pages themselves:

• A noindex meta robots tag or X-Robots-Tag header keeps a page out of search indexes, and AI search features that rely on those indexes.
• Redirect chains and loops waste crawl attempts. My redirect and HTTP status checker shows every hop and can request the page as GPTBot or Googlebot.
• Login walls and cookie walls that cover the content.

What access doesn't do

Being crawlable is the entry ticket, not a ranking factor. Once the bots can reach and read your pages, whether you get cited depends on the content itself: clear answers, specific facts and a site that's recognised as an authority on the topic. My guide on how to rank in ChatGPT covers that side. You may also see advice to add an llms.txt file; my llms.txt guide explains what it does and doesn't do.

Quick checklist

• robots.txt allows OAI-SearchBot, Claude-SearchBot and PerplexityBot on every subdomain you want cited
• Training bots blocked or allowed by choice, not by accident
• CDN, firewall and security plugin allow the search bots
• Main content present in the raw HTML
• Logs show the bots visiting, with 200 responses
• No unintended noindex, redirect chains or walls on key pages