An XML sitemap is a direct, unambiguous statement to search engines: "these are the URLs I want you to index." Every entry that turns out to be a redirect, an error, a noindexed page, or a URL nobody could even fetch weakens that statement, and enough of them teach a crawler to trust the rest of the file less.
What this finding means
Crawl Cove's sitemap-non-indexable-url check compares every URL your sitemap advertises against what the crawl actually found there. It fires when a listed URL does not resolve to a live, indexable, HTTP 200 page: it redirects, returns a non-200 status, is marked noindex, or could not be crawled at all (robots.txt disallowed it, or the request failed).
Real findings look like this, depending on the cause:
URL is listed in the XML sitemap but redirects to https://example.com/new-page; list the destination instead
URL is listed in the XML sitemap but returns HTTP 404
URL is listed in the XML sitemap but is marked noindex
URL is listed in the XML sitemap but was not crawled: robots.txt disallows it or it was unreachable
A fifth, more specific case fires when the sitemap lists a URL on a scheme the site does not actually serve: for example the sitemap says http:// but the site only answers on https://. Crawl Cove has still found the real page (at the scheme it does serve), so this is reported as a scheme mismatch in the sitemap rather than as a redirect the page itself doesn't have.
This check only runs on a full site audit. A crawl scoped by include/exclude patterns, or capped by max pages or depth, can legitimately miss a sitemap URL that was simply out of scope, so on a partial crawl the comparison is skipped rather than risk a false "not crawled" finding.
Why it matters
A sitemap is the one file where you get to say, in plain terms, exactly which URLs deserve a crawl and an index slot. Listing URLs that redirect, error, or carry a noindex directive muddies that signal. Search engines that repeatedly fetch dead weight from your sitemap have less reason to treat the rest of it as reliable, and every request spent on a URL that was never going to be indexed is a request not spent discovering a page that should be.
How to fix it
- Read the evidence for each flagged URL: it names the exact reason (redirect, status code, noindex, uncrawlable, or scheme mismatch), so the fix is different for each.
- Redirecting URLs: update the sitemap to list the final destination, not the URL that hops to it.
- 404s and other error statuses: remove the URL from the sitemap. If the content moved, list the new URL instead.
- Noindexed URLs: decide which is wrong. If the page should be indexable, remove the noindex directive; if it genuinely shouldn't be indexed, remove it from the sitemap instead.
- Uncrawlable URLs: check whether robots.txt is blocking a path your sitemap advertises; that combination sends search engines contradictory instructions.
- Scheme mismatches: regenerate the sitemap so every URL uses the scheme the site actually serves.
- Crawl Cove's free XML Sitemap Checker tool will surface every mismatch in one pass if you want a quick read before a full re-crawl.
- Re-crawl to confirm the sitemap and the live site now agree.
False positives and edge cases
- Off-host sitemap entries are never flagged as "not crawled". If a sitemap lists a URL on a different domain entirely, that is outside what this crawl was ever going to fetch, and the check leaves it alone rather than guessing.
- A scheme mismatch is reported as exactly that, not as a redirect. The crawl found the real page on the scheme the site does serve; the finding says the sitemap names the wrong scheme, not that the page itself redirects.
- This check only runs on a full audit. A scoped or page-capped crawl is not compared against the sitemap at all, since an out-of-scope URL would otherwise look identical to a genuinely broken one.
Related reading
For the reverse problem, a real, indexable page that the sitemap fails to mention at all, see Pages Missing From Your XML Sitemap.
Frequently asked questions
- Should a sitemap ever list a redirect?
- No. List the destination URL instead of the URL that redirects to it. A sitemap entry that redirects makes a search engine follow the hop before it reaches the page you actually want indexed, which is exactly the crawl-budget waste sitemaps exist to avoid.
- What if the sitemap URL was never crawled at all?
- The check reports this separately from a redirect or an error, because it is a different fact: either robots.txt disallowed the URL or the page was unreachable during the crawl. Either way, a sitemap that lists a URL search engines cannot fetch is telling them something untrue.
- Does this check run on every crawl?
- No, only on a full site audit. On a scoped or capped crawl, a sitemap URL can be legitimately outside the crawl's scope, so a partial crawl leaves this check silent rather than risk a false "not crawled" finding.
- My sitemap lists the http:// version but my site only serves https. Is that the same problem?
- It is a related but distinct case, reported with its own evidence: the sitemap is naming a URL on a scheme the site does not serve, even though the page itself is fine. The fix is the same in spirit (correct the sitemap), but the finding says so rather than wrongly claiming the page redirects.