Skip to content

Search Engine Spider Simulator

This spider simulator fetches one page as Googlebot Smartphone and shows you what a crawler gets from the HTML the server sends, before any JavaScript runs.

Fetch the page as Fetch as

Try it on or paste any public URL.

Note

We request the URL from our server, follow each redirect ourselves and read the HTML and response headers exactly as the server sent them, before any JavaScript runs. A request with the Googlebot user agent from our server is not from Google, so a site that checks where Googlebot comes from may answer it differently. Nothing from the page is stored.

At the top is the crawl and index verdict: the robots.txt line that applies to Googlebot on this URL, the meta robots and meta googlebot tags, the X-Robots-Tag header and the canonical. Below it is the page itself, stripped to what a crawler reads: the title, meta description, language, hreflang and structured data types, the H1 to H6 outline, the body text with a word count, every link marked internal or external and followed or not, and the images with and without alt text. When the HTML is an empty app shell or holds little text beside its scripts, it says so, because a crawler that does not run JavaScript sees only what is shown here. No sign-up and no email.

Use this tool often? Bookmark this page, so it is one click away next time.

Email me this result

Want a copy for your records or a client? Leave your email and we'll send you this result. It is written by a real person at Crawl Cove, not an automated blast, so it may take a little while to arrive. We'll only use your address to send it, unless you also tick the box below. Privacy policy.

That email address doesn't look right. Please check it and try again.

Something went wrong saving your address. Your result above is unaffected; please try again.

Got it. We'll email your result to you. Nothing else lands in your inbox unless you ticked the tips box.

How to view a page as a search engine spider

  1. Paste the URL

    Enter any public address, with or without https://. Redirects are followed hop by hop, and the page you end up on is the one shown.

  2. Choose who fetches it

    Googlebot Smartphone is the default, since Google indexes the mobile version of a page. Switch to a desktop browser to see whether the site sends crawlers something different.

  3. Read the verdict first

    Whether robots.txt lets Googlebot crawl the URL, and whether a robots tag, header or canonical keeps it out of the index. If the page cannot be crawled, the rest of the view is what Google would have seen, not what it sees.

  4. Read the page as text

    Check the headings make an outline, the main copy is in the body text, the links you need are real links, and the JavaScript note is clear. Copy the text or share the result link.

What this spider simulator shows

The fetch: the final URL, every redirect hop with its status code, the content type, the size of the HTML and how long the server took to start answering. The directives: the robots.txt group and rule that apply to Googlebot on this path, each meta robots, meta googlebot and X-Robots-Tag directive with where it came from, the canonical, and a verdict on whether Google may index the page. The head: the title, the meta description, the lang attribute, how many hreflang alternates the page declares and the structured data types in its JSON-LD, microdata or RDFa. Then the page as a crawler reads it: the H1 to H6 outline in source order, the body text with scripts, styles and noscript blocks left out, the word count, the links with internal, external and nofollow counts, and the images, split into those with alt text, those marked decorative with alt="" and those with no alt attribute at all.

check the H1 and heading order · check every link for nofollow

How Googlebot reads a page

Google describes three stages: crawling, rendering and indexing. Googlebot first fetches the URL and reads the HTML the server returns. It checks robots.txt before it fetches, and it finds new URLs from links written as <a> elements with an href. Pages that return a 200 are then queued for rendering, where a headless Chromium runs the JavaScript, and the rendered page is what gets indexed. The queue means rendering can happen some time after the crawl, so anything that only exists after a script runs depends on that second step working. This tool shows the first stage: the HTML as fetched, which is also all that a crawler that never renders will ever see.

JavaScript SEO audit: raw vs rendered HTML

Crawling and indexing are two different questions

robots.txt controls crawling, not indexing. A Disallow rule stops Googlebot fetching the page, but the bare URL can still appear in results if other pages link to it, and because Google cannot fetch the page it never sees a noindex on it either. A noindex in the meta robots tag or the X-Robots-Tag header controls indexing, and Google can only obey it on a page it is allowed to crawl. The canonical is a third signal: it asks Google to index another URL in place of this one. The simulator reads all three and shows them together, so you can tell a page that is blocked from one that is crawled and then dropped.

check whether a page can be indexed · Indexed, though blocked by robots.txt

When the content needs JavaScript

Some sites send a near empty HTML page, often a single <div id="root"> or <div id="__next">, and build the content in the browser. Google can usually index those pages after rendering, but many other crawlers read the server HTML only, most AI crawlers included, so they see the shell. The simulator flags an empty app root, a body with little text beside several scripts, a thin page whose noscript block asks for JavaScript, and a page with no links in its HTML. Each warning states what is in the HTML, not a guess about the rendered page. Server-side rendering or pre-rendering the main copy, headings and links fixes most of it. A framework is not a problem on its own: a site built with React or Vue that renders on the server sends its full text in the HTML and gets no warning.

Content that depends on JavaScript · check AI crawler access in robots.txt

Googlebot user agents and what a site sends them

Googlebot crawls most sites as Googlebot Smartphone, because Google indexes the mobile version of a page, and the simulator sends that user agent unless you pick a desktop browser. A site can answer a crawler differently from a visitor, on purpose or through a plugin, a firewall or a cache, so comparing the two is a quick test. A request claiming to be Googlebot from our server is not from Google, and some sites check that and turn it away; when that happens the error says so and the browser option is the way round it. Google treats showing crawlers different content from visitors as cloaking.

compare response headers as Googlebot and a browser

Frequently asked questions

What is a spider simulator?
A tool that fetches a web page the way a search engine crawler does and shows you what it can read: the text, the links, the headings and the tags in the HTML, without the layout, images or scripts a browser adds. It answers what a bot sees when it visits the page.
Is this spider simulator free?
Yes. It is free, needs no sign-up and no email, and the only limit is a rate limit that stops one visitor sending hundreds of checks a minute.
Does it show the page exactly as Google sees it?
It shows the first thing Google sees: the HTML the server sends to a request with the Googlebot Smartphone user agent, before JavaScript runs. Google renders JavaScript later and indexes the rendered page, so on a site that builds its content in the browser Google may see more than this. For the rendered version of your own page, use the URL Inspection live test in Search Console.
Why is the text on my page missing?
Usually because a script adds it after the page loads, so it is not in the server HTML. The JavaScript note says when that looks likely. Other causes are text inside an iframe, which belongs to the framed page, and text drawn into an image or a canvas.
Does robots.txt stop a page being indexed?
No. robots.txt stops crawling. A blocked URL can still be indexed without its content if other pages link to it. To keep a page out of results, let it be crawled and add a noindex meta tag or X-Robots-Tag header.
Can search engines see hidden text?
Text hidden with CSS is still in the HTML, so a crawler reads it and it appears in the body text here. Whether Google gives it weight is another matter; Google treats text hidden to mislead searchers as spam, while content in tabs or accordions that a visitor can open is normal.
Do AI crawlers run JavaScript?
Many do not, and read the server HTML only, which is what this view shows. If your main copy, headings or links are missing here, those crawlers are likely missing them too.
Why does the simulator only count some links?
It counts <a> elements with an href that point to a web page, because those are the links Google follows. Same-page anchors, email and phone links, and click handlers that only work with JavaScript are not counted. The list shows the first 500 links; the counts cover all of them.
Can I check every page on my site this way?
Not here: this tool reads one URL at a time. Crawl Cove, the desktop crawler, reads every page it crawls from its HTML, flags thin and noindexed pages, and can render a sample to compare raw with rendered. It has a permanent Free plan.
Can I share the result?
Yes. After a check the page address includes the URL and user agent you chose, so copying it gives a link that runs the same check. Results are cached for ten minutes.

From one page to your whole site

This shows one page from its server HTML. Crawl Cove reads every page of a site the same way and can render a sample in a browser to compare, so a template that hides its content behind JavaScript shows up as a list of pages. See audit a JavaScript site, raw vs rendered or see how Crawl Cove crawls a whole site.

Download free