Free technical SEO tools for crawling and indexing
These free technical SEO tools check whether search engines can crawl a page and are allowed to index it. They read the status code and response headers, robots.txt rules, canonical and noindex tags, XML sitemaps, redirects, links, hreflang, page weight and the mobile layout. They suit site owners, developers and SEOs who need a quick answer about one URL, or a small sample of a site, without installing anything or creating an account.
Indexing
-
Noindex Checker
Whether Google may index a URL, and which tag, header or rule is stopping it.
Open tool → -
Canonical Checker
Every canonical a URL declares, followed to see whether the target is a page that can be indexed.
Open tool → -
HTTP Header Checker
The status code of each hop and every response header, including X-Robots-Tag.
Open tool → -
Sitemap Finder
Every sitemap a site has, from robots.txt, the usual addresses and the homepage, checked against robots.txt.
Open tool → -
Sitemap URL Extractor
Every URL in a sitemap or a site's sitemaps, indexes and .xml.gz followed, with a CSV download.
Open tool → -
XML Sitemap Validator
Which pages in a sitemap are broken, redirected, noindexed or canonicalised elsewhere.
Open tool → -
XML Sitemap Generator
Crawl up to 500 URLs into a sitemap.xml, with every left-out URL and its reason.
Open tool →
Crawling and redirects
-
Spider Simulator
One page as a crawler reads its HTML: robots rules, head, headings, text, links and a JavaScript warning.
Open tool → -
Robots.txt Tester
Fetch, edit and test a robots.txt, and see whether a URL is allowed for each crawler.
Open tool → -
AI Crawler Access Checker
Whether robots.txt blocks GPTBot, ClaudeBot, PerplexityBot and other AI crawlers.
Open tool → -
Redirect Checker
Every redirect hop and its status code, with loops and meta refresh redirects flagged.
Open tool → -
Soft 404 Checker
Whether made-up URLs answer 404 or a 200 page or catch-all redirect, plus soft 404 signs on one page.
Open tool → -
WWW Redirect Checker
Whether http, https, www and non-www all reach one URL in one permanent redirect.
Open tool → -
.htaccess Redirect Generator
Exact-match 301 rules for Apache or Nginx from a list of old and new URLs.
Open tool →
Links
-
Anchor Text Checker
The anchor text of every link on a page, with vague, empty and conflicting anchors flagged.
Open tool → -
Broken Link Checker
Every link on a page checked live, with the status code of each broken one.
Open tool → -
Free Internal Link Checker
Click depth, inlinks and anchor text for a sample of pages crawled from a start URL.
Open tool → -
Nofollow Link Checker
Every link on a page with its rel and follow status, plus a nofollow set for the whole page.
Open tool → -
Free Orphan Page Checker
Which URLs in a sitemap no other checked page links to.
Open tool →
International
-
Hreflang Checker
Hreflang tags on a page or sitemap, with return links and x-default checked.
Open tool → -
Hreflang Tag Generator
A full hreflang tag set as HTML, sitemap entries or an HTTP header.
Open tool →
Performance and mobile
-
Page Size Checker
How much a page weighs, file by file, against Googlebot's 2 MB HTML limit.
Open tool → -
Mobile Friendly Test
The viewport tag, text size and fixed widths, and what Googlebot Smartphone is sent.
Open tool →
HTTPS
-
Mixed Content Checker
The http:// scripts, styles, images and forms on up to 25 HTTPS pages, with the fix for each.
Open tool →
Diagnose a page that will not index
Work from the server outwards. A page that answers with an error or a redirect cannot be indexed whatever its HTML says, so read the status code and headers first; an X-Robots-Tag noindex hides in the headers and never shows in view source. Then check the meta robots tag and robots.txt, which the noindex checker reads together. If those are clear, follow the canonical: a page that names another URL as canonical is asking Google to index that one. Last, make sure the page is in your sitemap and that the sitemap does not list it as something else.
Check a site before launch
Before a new site or redesign goes live, test the things that are hardest to spot by browsing it. Make sure robots.txt is not still blocking everything from staging. Run the broken link checker on the home page and the main templates, then crawl a sample with the internal link checker to see how deep the important pages sit. Finish with the mobile test and the page size checker on a heavy template, since Google indexes the mobile version and stops reading HTML past 2 MB.
Move pages or change URLs
Map every old URL to its new one and generate the redirect rules from that list, so each old address goes to its replacement in one hop. After the change is live, trace a few old URLs with the redirect checker to confirm a single 301 and no loop. Then validate the new sitemap so it lists only final URLs, and check for pages the move left with no internal links pointing at them.
Launch a site in more than one language
Generate the hreflang set once, with every language version and an x-default, and add the same set to each version of the page. Then run the hreflang checker on each one: every alternate must answer 200, be indexable and link back, or Google ignores the pair.
Further reading: Technical SEO audit checklist · A guide to technical SEO
Frequently asked questions
- What are technical SEO tools?
- Tools that check how a site is served and structured rather than what its pages say: status codes, response headers, robots.txt, canonical and noindex tags, sitemaps, redirects, internal links, hreflang, page weight and mobile rendering. These decide whether search engines can find, crawl and index a page at all, so they come before any work on content.
- Are these technical SEO tools free?
- Yes. Every tool on this page is free, with no sign-up and no email. The tools that look up a live URL fetch it from our server, because a browser is not allowed to read another site directly, and a rate limit stops one visitor sending hundreds of checks a minute.
- Which tool should I use first?
- Start with the HTTP header checker on the URL you are worried about. It shows the status code of every redirect and the final response, and any X-Robots-Tag header, which together rule out the most common reasons a page drops out of search. If the page should be indexed and is not, move on to the noindex checker.
- Can these tools check my whole website?
- No. Each one checks a page you enter, or a small sample of pages from the same site. To run the same kinds of check across every page, use a crawler. Crawl Cove has a free plan that runs on your own computer, with no account or card needed.
- Do they work on a staging or local site?
- The tools that fetch a URL only request public addresses, so a password-protected staging site or a local development server cannot be checked with them. The generators work with anything you type. A desktop crawler such as Crawl Cove can reach a staging site your own computer can reach.
- What is the difference between technical and on-page SEO tools?
- Technical tools check whether a page can be crawled and indexed. On-page tools check what the page says once it is indexed: the title, description, headings, content, images and structured data. A page needs both, and technical problems usually come first because they stop the on-page work from counting.
From one page to your whole site
These tools check one URL or a small sample. Crawl Cove reads the status codes, headers, canonicals, redirects and links of every page it crawls, from your own computer. Free plan, no account or card needed.