Some pages should never appear in search results, and noindex is the instruction that keeps them out. The trouble is that it works exactly the same way whether it was set on purpose or left on by accident, so the only way to tell the two apart is to check every page that carries it.
What this finding means
Crawl Cove's robots-noindex check flags any page excluded from search by a noindex directive (or its synonym none), reported through either of the two places that instruction can live:
- The
<meta name="robots">tag, which lives in the page's own HTML and is set by a theme, page builder or SEO plugin. - The
X-Robots-Tagresponse header, which is set by the server or CDN and is invisible in "view source": it will never show up no matter how carefully you read the page's HTML.
A real finding names exactly which mechanism is responsible, because they are fixed in completely different places:
<meta name="robots" content="noindex">
or
X-Robots-Tag: noindex
A page carrying both is reported once, not twice, since the underlying finding is the same either way: this page is excluded from search.
Why it matters
A noindex directive removes the page from search results entirely, with no partial state: Google will not rank it at all while the directive is present. That is correct for some pages and a serious, often invisible problem on one that should be ranking, and it is a very common accident after a staging or maintenance-mode site goes live with the directive still switched on.
How to fix it
- Decide whether the exclusion is actually intended. Not every noindex finding is a bug: see the false positives section below.
- If it came from the meta tag, find the setting in your CMS or SEO plugin ("Discourage search engines", "Hide from search results", or similar) and switch it off for that page, or edit the tag directly in the template if it was hand-coded.
- If it came from
X-Robots-Tag, the fix is in server or CDN configuration, not the page HTML:
# Apache (.htaccess): remove or comment out a line like
Header set X-Robots-Tag "noindex"
# Nginx: remove or comment out a line like
add_header X-Robots-Tag "noindex" always;
- If both are present, remove both. Removing only one leaves the page excluded, since either mechanism is sufficient on its own.
- Re-crawl the page to confirm the directive is gone, then check Google Search Console's URL Inspection tool to confirm the page is eligible for indexing again.
Heads up
On WordPress specifically, check Settings → Reading → "Discourage search engines from indexing this site" first. That single checkbox adds a sitewide noindex directive, and it is commonly left ticked by mistake after a site moves from staging to live.
False positives and edge cases
- A noindex tag is correct on plenty of pages, including thank-you pages, internal search results, admin or account screens, staging copies, and thin tag or filter archives that add nothing worth ranking. This finding is a "confirm this is intentional" prompt, not an automatic error.
- The two mechanisms are genuinely independent. A page can carry a header-based noindex with a completely clean-looking
<head>, so checking "view source" alone is not enough to rule this finding out. - This check does not judge whether the exclusion is a good idea, only that it exists and names where it comes from. The judgement call is yours.
Related reading
For the related fault where an indexable page's own canonical tag points at one of these excluded pages, see Canonical Pointing to a Noindex Page.
Frequently asked questions
- What is the difference between noindex and disallowing a page in robots.txt?
- robots.txt only tells a crawler not to fetch a URL; it does not stop that URL appearing in search results if something else links to it. noindex is fetched normally and tells the search engine not to list the result, which is the stronger and correct tool for keeping a specific page out of search.
- Why would a live site have noindex pages at all?
- Plenty of pages should never rank on their own: thank-you pages, internal search results, admin screens, staging copies, and thin tag or filter archives are all commonly and correctly excluded this way. The finding exists to confirm each one is intentional, not to say every noindex is wrong.
- I removed the meta tag but the page still doesn't rank. Why?
- Check the X-Robots-Tag response header separately. A page can carry the header without any meta tag in its HTML, so removing a tag that was never the actual source changes nothing.
- Does WordPress ever add this automatically?
- Yes. Settings → Reading → "Discourage search engines from indexing this site" adds a sitewide noindex directive the moment it is ticked, and it is a common accident to leave switched on after a staging site goes live.