Blocked by Robots.txt in Search Console

"Blocked by robots.txt" is the one status in the Page indexing report that can describe either a site working perfectly or a site that has made itself invisible. The report cannot tell the difference, because in both cases it is reporting the same thing: a rule in your robots.txt matched the URL and Google did not fetch it. This post covers what the block does and does not do, which URLs belong under it, how it goes wrong, and how to see every blocked URL on your site in one pass.

What the status means

Google's definition, from its Page indexing report documentation, is one line:

"This page was blocked by your site's robots.txt file."

Google discovered the URL, through a link, a sitemap or an earlier crawl, and queued it. Before fetching, it checked your robots.txt, found a Disallow rule that matched, and stopped. Google has not read the page. It does not know the title, the content, the canonical or whether there is a noindex on it. It knows only that the URL exists and that you told it not to look.

What a robots.txt block actually does

This is where most of the confusion comes from. A Disallow rule controls crawling. It does not control indexing. The two are separate steps, and robots.txt only reaches the first one.

That has two consequences people do not expect:

• A blocked URL can still be indexed. If enough pages link to it, Google may index the URL with no content, showing a bare address in results with a note that no description is available. The report has a separate status for that: indexed though blocked by robots.txt.
• A blocked URL cannot be noindexed. A noindex directive lives in the page's HTML or headers, and Google cannot read either without fetching the page. Blocking a page you have also noindexed hides the noindex. If the goal is to keep a page out of search, the page must be crawlable, and the excluded by noindex tag status is what that looks like when it works.

Robots.txt is a crawl-budget tool and a politeness tool. It is not a privacy tool and it is not an indexing tool.

When the block is correct

Most sites should have URLs under this status. Ours would, if Google had tried to fetch them: our robots.txt disallows /admin, /dashboard, /checkout, /profile and a few similar paths. Those are private areas with nothing to rank, and keeping Googlebot out of them costs nothing.

URLs that belong in this bucket:

• Admin, login and account areas
• Internal search result pages, which can generate an unbounded number of URLs
• Faceted navigation and filter combinations on e-commerce sites
• Calendar, tag and date archives that create crawl traps
• Cart, checkout and session-bound pages
• API endpoints and tracking pixels
• Staging and development paths on the production host

If every URL under the status is on that list, there is nothing to do, and a large count is not a problem.

When it is a real problem

The status needs urgent attention when a page you want ranking is on the list, because unlike most statuses this one means Google has not even read the page. Nothing you do to the page itself can help until the block is lifted. The common causes:

• A rule written too broadly. Disallow: /blog blocks /blog-archive and /blogging-tips, not just /blog/. Disallow: /*? blocks every URL with a parameter, including paginated listings and canonical product URLs on some platforms. Patterns match prefixes, and a missing trailing slash or an over-eager wildcard takes out far more than intended.
• A staging file that shipped. Staging sites are routinely blocked with Disallow: /. When the site goes live with the same file, every page disappears from the crawl queue at once.
• A CMS setting. WordPress's "Discourage search engines from indexing this site" and equivalents on other platforms write a blanket disallow. It is one checkbox, and it survives migrations.
• Blocked assets. Disallowing /wp-content/, /assets/ or /static/ stops Google fetching the CSS and JavaScript the page needs to render. The page itself is crawlable, but Google sees a broken version of it. Google's guidance is explicit that render-critical resources must be crawlable.
• A rule aimed at one bot that catches all of them. A User-agent: * group placed after a specific group, or a group with a typo in the bot name, can apply to Googlebot when it was meant for a scraper.

The quickest tell is the count. A site that suddenly has hundreds or thousands of URLs under this status, or whose indexed count drops at the same time the blocked count rises, has one of these.

How to fix it

1. Confirm the block. Run a live test on the URL in URL Inspection. The result reports whether robots.txt allowed the fetch. If it says blocked, the status is current.
2. Find the rule. Open /robots.txt on the live host, not a copy in your repository, and test the URL against it. The free robots.txt tester reports which rule matched for a given user-agent.
3. Narrow or remove the rule. Add the trailing slash, drop the wildcard, or delete the staging block. Keep the rules that are doing real work.
4. Check for a noindex you now need. If the page was blocked to keep it out of search, unblocking it means Google will crawl and may index it. Add a noindex before you lift the block, not after.
5. Request indexing. For a handful of important pages, request indexing in URL Inspection. For a site-wide unblock, resubmitting the sitemap is enough; Google will work through the queue.
6. Watch the status. Blocked URLs move to discovered currently not indexed first, then get crawled on Google's schedule.

The CSS and JavaScript case

One variant deserves its own mention because the page does not appear under this status at all. When robots.txt blocks a stylesheet or script directory, the HTML is still fetched, so the page shows as indexed. But Google renders pages, and a render that cannot load its CSS or JavaScript can miss content, fail mobile-friendliness or lose layout entirely. The symptom is a page that is indexed but ranks far below where it should, or that URL Inspection's rendered screenshot shows as a wall of unstyled text.

Google's guidance is to allow crawling of every resource the page needs to render. If a directory has to be blocked, allow the specific files or sub-paths the templates use.

Finding every blocked URL with a crawl

Search Console lists the blocked URLs Google happened to queue. It does not show the full set of pages your rules keep out, and it says nothing about a block you deployed this morning until Google next tries those URLs.

Crawl Cove fetches your robots.txt before anything else and obeys it, which makes it a direct test of the file. Every URL the crawl discovers and does not fetch because a rule matched is counted in the crawl's coverage summary, so after one crawl you can see how many URLs the site's own robots.txt kept out of the audit, and the same rule keeps search engines out of them too. The site-level robots-blocks-all-crawlers check fires at Critical severity when the file disallows Googlebot or Bingbot from the root with nothing re-permitting it, which is the staging-file-shipped case caught in seconds. For a site where you need to audit the blocked pages anyway, the New Crawl form has an Ignore robots.txt option, so you can crawl what the rules hide and see what the file is protecting.

The Live Index Status panel calls Google's URL Inspection API for any page you pick and reports the robots.txt state alongside the coverage state and last crawl date, so you can confirm Google sees the fix. Run a full SEO audit to see how much of the site your robots.txt is holding back.

Wrap-up

"Blocked by robots.txt" means Google never read the page. For private areas and crawl traps that is the point. For anything you want ranking it is the first thing to fix, because no other change can reach Google until the block is gone. Test the live file, narrow the rule, add a noindex first if the page should stay out of search, and remember that robots.txt controls crawling, not indexing. A crawl that obeys the file shows you exactly what it keeps out.