AI Crawlers in robots.txt: Who to Allow and Block

"Should I block AI crawlers in robots.txt?" is really three decisions, not one. AI crawlers do three different jobs (model training, AI search, and live user-triggered fetches), and blocking each one costs you something different. This guide shows how the bots divide up, what every rule really costs, and how to write robots.txt so the outcome is the one you intended.

The three purposes, and why they matter

Every AI crawler you will meet is doing one of three jobs. Knowing which one is the whole game.

The asymmetry is the point. Blocking training bots costs you nothing in traffic. Blocking search bots costs you the traffic. They are frequently conflated, and a robots.txt written to "keep our content out of AI" often quietly removes the site from AI answers as well.

Here is how the major tokens sort:

Tokens and behaviour change; each operator publishes its own list, and OpenAI's, Anthropic's, Google's and Perplexity's are the ones worth re-reading occasionally. This table was checked in August 2026.

How the rules are actually resolved

Two mechanics catch people out, and both silently produce the opposite of what was intended.

A bot-specific group overrides the wildcard group completely. Crawlers do not merge rules. If a bot finds a User-agent group naming it, that group is the only one it obeys. The User-agent: * group is ignored entirely for that bot. So this:

User-agent: *
Disallow: /private/

User-agent: GPTBot
Disallow: /drafts/

does not keep GPTBot out of /private/. GPTBot matches its own group, obeys only Disallow: /drafts/, and crawls /private/ freely. If a rule should apply to a named bot, it has to be repeated in that bot's group.

Disallow: with an empty value means "allow everything". It is not a no-op and it is not a comment. Disallow: / blocks the whole site; Disallow: on its own opens it.

Three configurations that make sense

Pick the one matching your actual goal, rather than assembling rules bot by bot.

1. Open to everything. The default, and the right answer for most marketing sites and publishers who want citations. No AI rules at all, or an explicit empty disallow to make the intent legible to whoever reads the file next.

2. Out of training, into answers. The most common considered position: you would rather not fund model training with your archive, but you very much want to be cited and clicked.

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: Applebot-Extended
Disallow: /

User-agent: Meta-ExternalAgent
Disallow: /

User-agent: Amazonbot
Disallow: /

Note what is absent: OAI-SearchBot, Claude-SearchBot and PerplexityBot are all still allowed, so your pages can still be surfaced and linked in AI answers.

3. Out of everything. For sites where the content is the product: paywalled journalism, proprietary research, member-only reference material. Add the search and user tokens to the list above. Understand the trade: you will not appear in AI answers, and a user who pastes your URL into an assistant will get an error.

Whichever you choose, keep your Sitemap: directive in the file. It is unaffected by any of this. Our free robots.txt generator builds a valid file from whichever configuration you pick, rather than you hand-typing user-agent blocks.

Checking what your rules actually do

Reading your own robots.txt is unreliable, for the reason above: the file as written and the file as resolved by a particular bot are different documents. The only trustworthy check is per-bot.

Crawl Cove's AI crawler access checker is a free browser tool that reads any site's live robots.txt and gives you a verdict for each bot in the table above, along with the exact rule that produced it. It is the quickest way to confirm that a rule you wrote for GPTBot is not being quietly overridden, and that a search bot you meant to allow has not been swept up by a wildcard. For the full, maintained list of tokens with what each one is for, see the AI crawler user-agents list. If you want the access verdict for one page together with the other things an answer engine checks (noindex and nosnippet rules, structure, author and date markup, /llms.txt), the free AI search visibility checker does that in one pass.

For a whole site, Crawl Cove's AI search visibility checker reports the same access question as findings rather than as a lookup: it flags when GPTBot, ClaudeBot, PerplexityBot or Google-Extended are disallowed from your site root, so an unintended block shows up next to your other technical issues instead of waiting to be noticed. It also raises a low-severity advisory when there is no /llms.txt file. The crawl runs locally on your own machine, so you can re-run it after every robots.txt edit and compare audits to confirm the change did what you meant. Try it free for 14 days.

Related reading

• How to write a robots.txt file covers the general syntax, rule resolution and common mistakes this post assumes.
• What is llms.txt? covers the curation side of the same question.
• Technical SEO: the complete guide shows where robots.txt sits among the rest.
• Crawl budget explained looks at how crawler access and crawl economics interact.

Wrap-up

AI crawler policy is three decisions wearing one coat. Training bots cost you nothing in traffic when you block them; AI search bots cost you citations and clicks; user-triggered bots cost you the visitor who asked about you by name. Decide those separately, remember that a bot-specific group replaces the wildcard group rather than adding to it, and then verify the file per bot rather than by eye. The rules are easy. Confirming they resolve the way you meant is the part worth the ten minutes.