Skip to content

AI Crawler Access Checker

Enter any website and see, bot by bot, whether its robots.txt lets AI crawlers in or keeps them out — for model training, AI search indexing and user-triggered fetches. We read the live robots.txt and show the exact rule that decides each verdict.

Try:

Note

We fetch the site’s live /robots.txt and evaluate each crawler against the root path (/) using the same longest-match rules search engines use. robots.txt is a request, not a wall — well-behaved AI crawlers honour it, but it doesn’t technically prevent access. Google-Extended and Applebot-Extended only govern AI training, not search crawling.

Email me this result

Want a copy for your records or a client? Leave your email and we'll send you this result — written by a real person at Crawl Cove, not an automated blast, so it may take a little while to arrive. We'll only use your address to send it, unless you also tick the box below. Privacy policy.

That email address doesn't look right — please check it and try again.

Something went wrong saving your address — your result above is unaffected; please try again.

Got it — we'll email your result to you. Nothing else lands in your inbox unless you ticked the tips box.

Which AI crawlers this checks

It tests a site’s robots.txt against the crawlers that matter — GPTBot (OpenAI), ClaudeBot (Anthropic), Google-Extended, PerplexityBot, Applebot-Extended, Bytespider, CCBot and more — across model training, AI search indexing and user-triggered fetches, and shows each bot’s verdict with the exact rule that produced it.

Should you block AI crawlers?

There’s no single right answer. Blocking training bots like GPTBot and Google-Extended keeps your content out of model training. But blocking AI search bots like OAI-SearchBot and PerplexityBot can cost you citations and referral traffic in AI answers. This checker shows exactly where a site stands so you can decide deliberately.

Frequently asked questions

How do I stop ChatGPT from using my website?
Disallow GPTBot in your robots.txt to opt out of OpenAI model training, and disallow OAI-SearchBot and ChatGPT-User to keep pages out of ChatGPT’s search and live browsing. This tool confirms whether those rules are working.
Does Google-Extended block Google Search?
No. Google-Extended only controls whether your content trains Google’s AI models (Gemini/Vertex). Googlebot still crawls your site for Search, and your rankings are unaffected.
Does blocking AI crawlers in robots.txt actually stop them?
Reputable crawlers honour robots.txt, but it’s a request, not enforcement — it can’t technically prevent a bot that ignores it. It remains the standard, expected way to state your preference.
What’s the difference between GPTBot and OAI-SearchBot?
GPTBot crawls the web to train OpenAI’s models. OAI-SearchBot indexes pages so ChatGPT can surface and cite them in search. You can allow one and block the other.

From one page to your whole site

Controlling AI crawlers is one page in robots.txt; getting it right across a whole site is harder. Crawl Cove audits AI crawler access and AI search visibility (GEO) sitewide, so you can see — and fix — exactly who can reach your content. See the features or compare the plans.

Download Crawl Cove

More free tools