How Many UK Websites Block AI Crawlers?

About one in four of the UK's biggest websites block the AI training crawlers in robots.txt. We read the robots.txt files of the 1,000 most visited .co.uk sites on 7 October 2026: of the 617 we could read, 23.2% block OpenAI's GPTBot, 22.5% block Anthropic's ClaudeBot and 21.6% block Common Crawl's CCBot.

That is the number most surveys stop at. Two findings from the rest of the test matter more if you run a site:

• Most sites that block AI training also block AI search. 71% of the sites that block GPTBot also block OAI-SearchBot, the crawler that decides whether ChatGPT can cite them.
• robots.txt is often not the real policy. When we requested each homepage as each AI crawler, 21.5% of the sites we could test refused at the firewall a crawler their own robots.txt allows. Another 40 sites did not refuse at all: they asked for payment.

How many websites block AI crawlers in robots.txt

We fetched every site's robots.txt and evaluated it for 23 AI and search crawler tokens with the same parser our AI Crawler Access Checker uses (RFC 9309 rules: most specific group wins, groups naming the same crawler are merged, longest matching rule wins). "Blocked" below means the file disallows the root path, /, for that crawler.

Base: 617 robots.txt files we could read, out of 1,000 sites. A third of the files (200, or 32.4%) name at least one AI crawler by its token; the rest rely on their User-agent: * rules.

Two patterns stand out.

Google-Extended is blocked half as often as GPTBot (11.8% against 23.2%). Google-Extended governs whether content trains Google's Gemini models and whether Gemini and Vertex AI can ground answers in it. It has no effect on Google Search. Sites appear to be far more cautious about anything carrying Google's name, even though blocking it costs nothing in Search.

The training and search crawlers are mostly blocked together. OpenAI, Anthropic and Perplexity all run separate crawlers for training, for their AI search index and for fetching a page a user has asked about, precisely so a site can opt out of one without the others. Few sites use that split:

• Of the 143 sites blocking GPTBot, only 41 (28.7%) leave OAI-SearchBot allowed. The other 102 are out of ChatGPT search as well as OpenAI's training data.
• Of the 139 blocking ClaudeBot, 74 (53.2%) still allow Claude-User, so a person who pastes one of their links into Claude gets the page read. The rest refuse that too.

If your goal is "do not train on my content, but do cite me", the rule is the one in our guide to AI crawlers in robots.txt: disallow the training tokens by name and leave the search and user-triggered ones alone.

What the firewall says

robots.txt is a request. A CDN or firewall can refuse a crawler on its user agent before robots.txt is ever read, and Cloudflare, for one, offers a one-click setting to block AI bots. So for every site that answered a normal browser request, we requested the homepage again as 11 AI crawlers that fetch pages, and compared the answers.

Three controls keep the result honest:

• A browser request, twice. A site that is down or flaky is not mistaken for a block.
• A made-up crawler (FooBot). A site that refuses every unknown bot is not counted as targeting AI.
• A request claiming to be Googlebot. Nobody blocks the real Googlebot on purpose, but 60 of the 558 sites we could test (10.8%) refused our fake one. Those sites check crawlers by IP address, so our requests looked like impostors whatever name they carried. We left all 60 out of the firewall figures below, which is why this test counts sites such as Barclays and Airbnb as unknown rather than as blocking AI.

We also sent each request to a fresh URL, so a CDN cache could not answer for the firewall, and dropped any response that still came from cache.

Base: 498 sites that let our browser, our made-up crawler and our fake Googlebot through.

127 of those 498 sites (25.5%) refuse at least one AI crawler at the firewall, and 107 (21.5%) refuse one that their own robots.txt allows. Rightmove is a clean example: its robots.txt does not mention Bytespider at all, yet a request with Bytespider's user agent gets a 403 while our fake Googlebot gets the page.

The firewalls are more selective than the robots.txt files. Training crawlers are refused far more often than the search and user-triggered fetchers (GPTBot 14.8% against OAI-SearchBot's 3.5%, ClaudeBot 14.9% against Claude-User's 2.6%), which suggests the firewall rules are mostly aimed at training. The gap cuts both ways, though: a site owner who reads their robots.txt and sees GPTBot allowed may not know their CDN has been refusing it all along.

Pay-per-crawl is already here

The most striking answers were not refusals. 40 sites asked the AI crawler to pay:

• 29 sites redirect AI crawlers to a tollbit. subdomain, TollBit's pay-per-crawl gateway, while sending browsers the normal page. They include the Mirror, the Daily Star, the Daily Record, the Manchester Evening News, WalesOnline, the Liverpool Echo, the Birmingham Mail, the Bristol Post and OK! magazine.
• 11 sites answer with HTTP 402 Payment Required, a status code HTTP has kept "reserved for future use" since the 1990s. They include the Independent, Autocar, Parkers, CAR magazine, Grazia, Stylist and reed.co.uk.

For an AI crawler that does not pay, these pages are as unreadable as a blocked one, and an assistant that cannot read a page cannot quote or cite it. Our AI Crawler Access Checker now reports this as "asked to pay" rather than as a refusal or an ordinary redirect.

llms.txt adoption

llms.txt is a proposed file that gives AI assistants a short, curated map of a site's best pages. 72 of the 839 sites we could reach (8.6%) serve one. That is low, but not negligible for a proposal that no major AI provider has publicly said it reads. Argos and Shein are among the sites that have one. Our guide to what llms.txt is covers whether it is worth adding, and the llms.txt generator builds one from your sitemap.

Method and limits

• Sample: the first 1,000 .co.uk domains in the Tranco list (ID 26Y29, downloaded 7 October 2026), a research ranking of the most visited domains that anyone can reproduce. We removed DNS and hosting infrastructure entries (awsdns-*, livedns, nominetdns). It covers every sector, from news and retail to banks, travel and job boards.
• When: every request was made on 7 October 2026, from one server in a German data centre, at a polite rate (about 17 requests per site, spread over several seconds).
• What we could not see: 241 of the 839 reachable sites (28.7%) refused even our browser request, because it came from a data centre rather than a home connection. That group includes many large retailers, so this is not a reliable picture of UK e-commerce, and we have not described it as one. 145 sites refused to serve us their robots.txt for the same reason; we counted those as unreadable, not as having no rules.
• The live test is not the bot's own network. Our requests carry each crawler's user agent but come from our IP address. We control for sites that verify crawlers by IP (the fake Googlebot test above), but a site could still treat the real crawler differently from our request in ways we cannot see.
• Homepage only. Rules for a specific directory, such as /search or /account, are not counted as blocking.

The per-site results are free to use with a link back to this page: download the CSV (1,000 rows: robots.txt verdict and live result per crawler, llms.txt, and whether the site verifies crawlers by IP).

Check your own site

The fastest way to find out which of these groups you are in is to test it. Enter your domain in the AI Crawler Access Checker for the robots.txt verdict per crawler, then press "Test live access" to run the same firewall test we used here. To see whether a specific page is ready to be cited once a crawler gets in, the AI search visibility checker scores it.