When robots.txt names a sitemap that answers 404, every crawler that follows the line is sent to a file that is not there. If it is the only sitemap the file names, those crawlers find no sitemap at all. The fix is one line, once you know where the sitemap lives now.
What this finding means
Crawl Cove's sitemap-declared-missing check reads every Sitemap: line in robots.txt during the crawl and fires when one of those addresses answers 404 or 410. Only the URLs robots.txt names are judged; sitemaps listed inside a sitemap index are not. A real finding reads like this:
robots.txt names the sitemap https://www.example.com/sitemap.xml (HTTP 404), which does not exist. No sitemap could be read, so a crawler following robots.txt finds none.
When robots.txt names several sitemaps and only some are missing, the evidence lists the dead ones (up to five, then a count) and says the others were read.
Why it matters
robots.txt is the one file every well-behaved crawler reads before it crawls, and the Sitemap: line is how it learns where your list of URLs is. A line pointing at a missing file spends that on nothing. Google and Bing may still have the right sitemap if you submitted it in Search Console or Bing Webmaster Tools, but every other crawler is relying on robots.txt, and so is any tool or engine you never submitted to. That is why this check is Medium, above the advisory sitemap findings: when the dead line is the only one, the sitemap is effectively invisible to them.
It is not rare. In our October 2026 sweep of small business websites, 22 of the 860 sites whose robots.txt names a sitemap (2.6%) named one that answers 404.
How to fix it
- Find where the sitemap is now. Try
/sitemap_index.xml,/sitemap.xmland, on WordPress,/wp-sitemap.xmlin a browser. Yoast SEO and Rank Math both serve/sitemap_index.xml; WordPress's own sitemap (since version 5.5) lives at/wp-sitemap.xmland is switched off when an SEO plugin provides one. - Find where robots.txt comes from. A physical
robots.txtfile at the site root overrides the one WordPress or an SEO plugin generates, so check for that file first. A line naming the old plugin's address in a hand-written file is the most common cause. - Point the line at the real file, with its full URL:
Sitemap: https://www.example.com/sitemap_index.xml. If the site has no sitemap any more, remove the line, or better, publish one. - If the sitemap should exist but 404s, the problem is the sitemap, not robots.txt: on WordPress that is usually rewrite rules (re-saving the Permalinks settings regenerates them) or a cache serving an old 404.
- Check it. Open the address from robots.txt in a browser and confirm it loads as XML, then run the free XML sitemap validator on it. A re-crawl in Crawl Cove clears the finding.
False positives and edge cases
- Refusals are not reported as missing. A 403, a 5xx or a timeout on the sitemap URL is often a firewall or rate limit refusing the crawler, so the check ignores those statuses rather than report a working sitemap as gone.
- Sitemaps inside an index are not judged here. A broken child sitemap listed in a working index does not trigger this finding.
- A sitemap that moved behind a redirect is fine for this check. Only a final 404 or 410 counts. Pointing the line at the final address is still tidier, because the protocol expects the sitemap's own URL.
Related reading
If robots.txt names no sitemap at all, the crawl reports a sitemap missing from robots.txt instead. For everything else the file controls, see how to write a robots.txt file.
Specification: Sitemaps XML format
Frequently asked questions
- Why would robots.txt name a sitemap that does not exist?
- Usually because the sitemap moved and robots.txt did not. The common causes are switching SEO plugin (each one serves its sitemap at its own address), a site migration that kept the old robots.txt, or a sitemap feature that was switched off while a hand-written robots.txt still names it.
- Does a 403 or a server error count as missing?
- No. Crawl Cove only reports a sitemap as missing when its address answers 404 or 410. A 403, a 5xx or a timeout is as often a firewall refusing the crawler as a broken file, so the check stays silent rather than tell you a working sitemap is gone.
- Is it a problem if robots.txt names two sitemaps and only one works?
- Less of one. The finding still fires, because the dead line is wrong, but its evidence says the other sitemaps were read, so crawlers that follow robots.txt still reach a sitemap. Remove or correct the dead line all the same.
- Should I just delete the Sitemap line?
- Only if the site no longer has a sitemap. If it has one at another address, point the line there instead; a correct Sitemap line is how crawlers you never submitted the sitemap to find it.
- I edited robots.txt in WordPress but the old line is still there. Why?
- A physical robots.txt file at the site root is served by the web server before WordPress sees the request, so it overrides the robots.txt that WordPress or your SEO plugin generates. Edit or delete that file, then check the live robots.txt in a browser.