Indexed, Though Blocked by Robots.txt

"Indexed, though blocked by robots.txt" is the status that teaches most people how robots.txt actually works. You blocked a URL, Google indexed it anyway, and the report is telling you both things at once. This post explains how that happens, why the listing looks the way it does, and the counter-intuitive fix: let Google in so it can read the instruction that keeps the page out.

What "Indexed, though blocked by robots.txt" means

Google's definition, from its Page indexing report documentation:

"The page was indexed despite being blocked by your website's robots.txt file."

Google never fetched the page. Your Disallow rule stopped the crawl, exactly as the blocked by robots.txt status describes. But Google also found links to the URL, from your own pages or from other sites, and decided on the strength of those links that the URL should be in the index. So it indexed what it had: the address, and whatever anchor text pointed at it.

The report lists this under the warning group rather than the error group, because the page is in the index. It is still a signal that something is set up wrong, because no one blocks a URL and also wants it listed.

What the listing looks like

A result indexed this way has no snippet, because Google has no content to draw one from. Depending on the query, the result shows the bare URL as its title, or a title assembled from the anchor text of links pointing at it, and a line saying no information is available for the page.

For a page you want ranking, that is a result nobody clicks. For a page you meant to hide, it is a listing you did not want, and one whose title you did not write. Neither outcome is what the block was for.

Why robots.txt cannot keep a URL out of the index

Robots.txt controls one thing: whether a crawler fetches a URL. Indexing is a separate decision that Google makes from every signal it has about a URL, and links are a signal. A blocked URL with inbound links is a URL Google knows about, has reason to think matters, and cannot read. Google's documented behaviour in that situation is to index the URL anyway.

The instruction that keeps a page out of the index is noindex, and it lives in the page's HTML or in an X-Robots-Tag response header. Google can only see it by fetching the page. Block the fetch, and the one instruction that would have worked becomes invisible.

That is the whole mechanism. Robots.txt and noindex are not two ways of doing the same thing. They are two tools that cancel each other out when used together.

When it is acceptable to leave

There is a narrow case where the status is tolerable: a URL space you block to protect crawl budget, where the occasional bare listing does no harm. Internal search results with a few stray external links are the usual example. The listing is useless but harmless, and unblocking a space designed to generate unbounded URLs costs more than the stray result.

Everywhere else, fix it.

How to fix it

The fix depends on what you want for the page.

If the page should rank

Remove the Disallow rule. Google crawls the page, reads its content, and the next evaluation replaces the bare listing with a real one. Check the page for a stray noindex first, because a page that was blocked for a while may have had one added by a plugin in the meantime. Then request indexing in URL Inspection.

If the page should be out of search

1. Remove the Disallow rule. This feels wrong. It is the required step, because Google has to fetch the page to see the next one.
2. Add a noindex directive. In the robots meta tag or the X-Robots-Tag header. Do this before or at the same time as unblocking, never after, so Google does not index the content in the gap.
3. Wait for Google to recrawl. The status moves to excluded by noindex tag once Google has fetched the page and seen the directive. Check the last crawl date in URL Inspection to know whether that has happened yet.
4. Then decide about the block. Once the page is out of the index, you can leave it crawlable with the noindex in place, which is the stable state. If you must block it again for crawl-budget reasons, understand that the moment Google's cached copy of the noindex ages out, the links can pull the URL back in.

If the page should not exist at all, return a 404 or 410 instead. A dead page needs no robots rule and no noindex; Google drops it after a few recrawls. The not found 404 in Search Console status is what that looks like.

Password-protected and private pages

A page that must stay private should not rely on either robots.txt or noindex, because both depend on every crawler reading and respecting a hint. Put it behind authentication so the server returns a 401 or 403 to anyone without credentials. Google then cannot read it, cannot index its content, and reports it under its own statuses; see blocked due to access forbidden (403) for what those look like and when they are the right answer.

Finding the links keeping a blocked URL in the index

The status tells you the URL is blocked and indexed. It does not tell you which of your own pages link to it, and on your own site that is the signal you control. Fifty internal links to a blocked URL are fifty reasons for Google to keep listing it.

Crawl Cove obeys robots.txt when it crawls, so every URL a Disallow rule keeps out of the crawl is counted in the run's coverage summary, which tells you at a glance how many URLs the file is hiding and that search engines are refused the same pages. The site-level robots-blocks-all-crawlers check fires at Critical when the file blocks Googlebot or Bingbot from the root, the case where this status appears for your most-linked pages at once. For the pages you decide to unblock and noindex, the robots-noindex check confirms the directive is present in the HTML or the X-Robots-Tag header once the crawl can reach them, and sitemap-non-indexable-url catches blocked or noindexed URLs still listed in your sitemap, which is one more signal telling Google the URL matters.

The Live Index Status panel calls the URL Inspection API for any page you choose and reports the robots.txt state, the indexing state and the last crawl date together, so you can see when Google has finally read the noindex. Run a full SEO audit to see every blocked URL, every internal link pointing at one, and every noindex, in a single crawl.

Wrap-up

"Indexed, though blocked by robots.txt" is what happens when robots.txt is asked to do a job it cannot do. Google indexed a URL it was not allowed to read, because links told it the URL mattered. The fix runs against instinct: unblock the page so Google can read a noindex, or unblock it so Google can read the content, depending on what you want. Blocking harder never works, because the page Google cannot read is the page Google cannot be told to drop.