Skip to content

Free technical SEO tools for crawling and indexing

These free technical SEO tools check whether search engines can crawl a page and are allowed to index it. They read the status code and response headers, robots.txt rules, canonical and noindex tags, XML sitemaps, redirects, links, hreflang, page weight and the mobile layout. They suit site owners, developers and SEOs who need a quick answer about one URL, or a small sample of a site, without installing anything or creating an account.

Indexing

  • Noindex Checker

    Whether Google may index a URL, and which tag, header or rule is stopping it.

    Open tool →
  • Canonical Checker

    Every canonical a URL declares, followed to see whether the target is a page that can be indexed.

    Open tool →
  • HTTP Header Checker

    The status code of each hop and every response header, including X-Robots-Tag.

    Open tool →
  • Sitemap Finder

    Every sitemap a site has, from robots.txt, the usual addresses and the homepage, checked against robots.txt.

    Open tool →
  • Sitemap URL Extractor

    Every URL in a sitemap or a site's sitemaps, indexes and .xml.gz followed, with a CSV download.

    Open tool →
  • XML Sitemap Validator

    Which pages in a sitemap are broken, redirected, noindexed or canonicalised elsewhere.

    Open tool →
  • XML Sitemap Generator

    Crawl up to 500 URLs into a sitemap.xml, with every left-out URL and its reason.

    Open tool →

Crawling and redirects

  • Spider Simulator

    One page as a crawler reads its HTML: robots rules, head, headings, text, links and a JavaScript warning.

    Open tool →
  • Robots.txt Tester

    Fetch, edit and test a robots.txt, and see whether a URL is allowed for each crawler.

    Open tool →
  • AI Crawler Access Checker

    Whether robots.txt blocks GPTBot, ClaudeBot, PerplexityBot and other AI crawlers.

    Open tool →
  • Redirect Checker

    Every redirect hop and its status code, with loops and meta refresh redirects flagged.

    Open tool →
  • Soft 404 Checker

    Whether made-up URLs answer 404 or a 200 page or catch-all redirect, plus soft 404 signs on one page.

    Open tool →
  • WWW Redirect Checker

    Whether http, https, www and non-www all reach one URL in one permanent redirect.

    Open tool →
  • .htaccess Redirect Generator

    Exact-match 301 rules for Apache or Nginx from a list of old and new URLs.

    Open tool →

Links

International

  • Hreflang Checker

    Hreflang tags on a page or sitemap, with return links and x-default checked.

    Open tool →
  • Hreflang Tag Generator

    A full hreflang tag set as HTML, sitemap entries or an HTTP header.

    Open tool →

Performance and mobile

  • Page Size Checker

    How much a page weighs, file by file, against Googlebot's 2 MB HTML limit.

    Open tool →
  • Mobile Friendly Test

    The viewport tag, text size and fixed widths, and what Googlebot Smartphone is sent.

    Open tool →

HTTPS

  • Mixed Content Checker

    The http:// scripts, styles, images and forms on up to 25 HTTPS pages, with the fix for each.

    Open tool →

Diagnose a page that will not index

Work from the server outwards. A page that answers with an error or a redirect cannot be indexed whatever its HTML says, so read the status code and headers first; an X-Robots-Tag noindex hides in the headers and never shows in view source. Then check the meta robots tag and robots.txt, which the noindex checker reads together. If those are clear, follow the canonical: a page that names another URL as canonical is asking Google to index that one. Last, make sure the page is in your sitemap and that the sitemap does not list it as something else.

  1. HTTP Header Checker
  2. Noindex Checker
  3. Canonical Checker
  4. Robots.txt Tester
  5. XML Sitemap Validator

Check a site before launch

Before a new site or redesign goes live, test the things that are hardest to spot by browsing it. Make sure robots.txt is not still blocking everything from staging. Run the broken link checker on the home page and the main templates, then crawl a sample with the internal link checker to see how deep the important pages sit. Finish with the mobile test and the page size checker on a heavy template, since Google indexes the mobile version and stops reading HTML past 2 MB.

  1. Robots.txt Tester
  2. Broken Link Checker
  3. Free Internal Link Checker
  4. Mobile Friendly Test
  5. Page Size Checker

Move pages or change URLs

Map every old URL to its new one and generate the redirect rules from that list, so each old address goes to its replacement in one hop. After the change is live, trace a few old URLs with the redirect checker to confirm a single 301 and no loop. Then validate the new sitemap so it lists only final URLs, and check for pages the move left with no internal links pointing at them.

  1. .htaccess Redirect Generator
  2. Redirect Checker
  3. XML Sitemap Validator
  4. Free Orphan Page Checker

Launch a site in more than one language

Generate the hreflang set once, with every language version and an x-default, and add the same set to each version of the page. Then run the hreflang checker on each one: every alternate must answer 200, be indexable and link back, or Google ignores the pair.

  1. Hreflang Tag Generator
  2. Hreflang Checker

Further reading: Technical SEO audit checklist · A guide to technical SEO

Frequently asked questions

What are technical SEO tools?
Tools that check how a site is served and structured rather than what its pages say: status codes, response headers, robots.txt, canonical and noindex tags, sitemaps, redirects, internal links, hreflang, page weight and mobile rendering. These decide whether search engines can find, crawl and index a page at all, so they come before any work on content.
Are these technical SEO tools free?
Yes. Every tool on this page is free, with no sign-up and no email. The tools that look up a live URL fetch it from our server, because a browser is not allowed to read another site directly, and a rate limit stops one visitor sending hundreds of checks a minute.
Which tool should I use first?
Start with the HTTP header checker on the URL you are worried about. It shows the status code of every redirect and the final response, and any X-Robots-Tag header, which together rule out the most common reasons a page drops out of search. If the page should be indexed and is not, move on to the noindex checker.
Can these tools check my whole website?
No. Each one checks a page you enter, or a small sample of pages from the same site. To run the same kinds of check across every page, use a crawler. Crawl Cove has a free plan that runs on your own computer, with no account or card needed.
Do they work on a staging or local site?
The tools that fetch a URL only request public addresses, so a password-protected staging site or a local development server cannot be checked with them. The generators work with anything you type. A desktop crawler such as Crawl Cove can reach a staging site your own computer can reach.
What is the difference between technical and on-page SEO tools?
Technical tools check whether a page can be crawled and indexed. On-page tools check what the page says once it is indexed: the title, description, headings, content, images and structured data. A page needs both, and technical problems usually come first because they stop the on-page work from counting.

From one page to your whole site

These tools check one URL or a small sample. Crawl Cove reads the status codes, headers, canonicals, redirects and links of every page it crawls, from your own computer. Free plan, no account or card needed.

Download free

All free SEO tools · On-page SEO tools