Every robots.txt has one line that can empty a site out of search: Disallow: / with nothing re-permitting it. It is the normal setting for a staging copy, and it ships to production more often than anyone admits. Crawl Cove's robots-blocks-all-crawlers check names it, because until it existed the only sign of the problem was a coverage line reading "robots-blocked", which looks like a setting rather than a disaster.
What this finding means
The check reads the site's robots.txt with the same parser and group-selection rules the crawlers use, then asks one question for Googlebot and one for Bingbot: does the group that applies to this crawler disallow the site root? A crawler uses the most specific group that names it, and only falls back to User-agent: * when none does, so the check follows the same order.
A real finding names the engines and the rule that blocks them. Through the wildcard:
User-agent: *
Disallow: /
Or through an engine-specific group:
User-agent: Googlebot
Disallow: /
One finding is raised per blocking group, so a file whose wildcard blocks both engines produces a single finding naming Google and Bing together. The finding is site-level and critical: nothing on the pages themselves is wrong, and nothing on them can be crawled.
Why it matters
A root disallow stops search engines fetching every page at once. New and changed pages are never indexed, and existing results decay as the index goes stale, which is why the drop in traffic tends to arrive weeks after the file changed and nobody connects the two. It is the single most expensive line a robots.txt can contain, and it leaves no trace on the pages a person looks at.
How to fix it
- Confirm the rule is not deliberate. A site that is meant to stay out of search entirely, a private staging copy, say, is the one legitimate case. For a public site, carry on.
- Open robots.txt at the site root (
/robots.txt) and find the group the finding names. If it saysUser-agent: *the wildcard is the culprit; if it names Googlebot or Bingbot, the fix is in that group. - Remove the
Disallow: /line, or narrow it to the paths you actually want hidden. A file that disallows nothing at all is valid:
User-agent: *
Disallow:
- Check for an engine-specific group that re-permits the root if you need the wildcard block to stay for other crawlers. An empty
Disallow:or an explicitAllow: /inside aUser-agent: Googlebotgroup lets Googlebot through even when*is blocked, and the check reads it that way. - Re-crawl, then use Google Search Console's robots.txt report to confirm the live file is the one you edited. A CDN or a deployment pipeline can serve a different robots.txt from the one in your repository.
Heads up
If this site was recently moved from staging, check the deployment itself as well as the file. Many hosts and frameworks generate a blocking robots.txt from an environment flag, so editing the file by hand is undone on the next deploy unless the flag is changed too.
False positives and edge cases
- A missing robots.txt is not a finding. A 404 means "allow all", and the check stays silent. A server error or an unreachable file is a crawl-coverage matter, not a claim about the file's contents, so the check stays silent there too rather than guess.
- A file that only disallows paths never triggers this.
Disallow: /wp-admin/orDisallow: /searchis normal crawl control and is not read as a root block. - An engine with its own permissive group is not blocked, even when the wildcard disallows the root. The check follows the crawlers' own precedence rules, so it will not report Googlebot blocked when a
User-agent: Googlebotgroup lets it in. - Crawl Cove's own crawl is not the evidence. Since version 1.3.0 a crawl can tick Ignore robots.txt for a staging site, and the check still reads the file as the search engines would. It reports what Google and Bing will do, not what the audit did.
Related reading
Write a file that says what you mean with the free robots.txt generator, which starts from allow-all defaults, and read How to write a robots.txt file for the group and precedence rules this check follows. For the AI crawlers that fall under the same wildcard, see AI crawlers and robots.txt. To keep a specific page out of search the right way, see Noindex: What It Means and How to Fix It. If the blocking file belongs to a staging site you need to audit anyway, How to crawl a staging site behind a login covers the Ignore robots.txt option.
Google documentation: Robots.txt introduction and guide
Frequently asked questions
- Will removing Disallow: / bring my rankings straight back?
- Crawling resumes as soon as search engines re-fetch robots.txt, which usually happens within a day, but pages are re-crawled and re-indexed over days or weeks after that. Removing the rule stops the damage; it does not undo it instantly, so fix it the moment you see it.
- My site is still showing in Google even though robots.txt blocks everything. Why?
- robots.txt only stops crawling. A URL that was already indexed stays indexed until it is crawled again, and Google can still list a blocked URL it has seen links to, with no description. That is why a blanket block looks harmless for a while and then rankings decay.
- Does Disallow: / under User-agent: * block AI crawlers too?
- Yes, any crawler that obeys robots.txt and has no group of its own falls under the wildcard. Crawl Cove reports the AI crawlers separately in its AI bot access check, so a site with this finding usually carries those findings as well.
- Is robots.txt the right way to keep a page out of search?
- No. Use a noindex directive on a page the crawler is allowed to reach. A disallowed page cannot be crawled, so a noindex on it is never seen, and the URL can still be listed. Reserve robots.txt for crawl control, not for hiding pages.