Search Engine Spider Simulator
This spider simulator fetches one page as Googlebot Smartphone and shows you what a crawler gets from the HTML the server sends, before any JavaScript runs.
Try it on or paste any public URL.
What a crawler is allowed to do
- robots.txt for Googlebot
- Robots tags and headers
-
None. With no meta robots, meta googlebot or X-Robots-Tag, the page is open to indexing and its links are followed.
- Canonical
- None declared.
The fetch
- Status
- Redirects
- HTML size
- First byte
This URL is not an HTML page, so there is no text, heading or link view. The verdict above still applies to it.
Google renders JavaScript after it crawls, so it may still index what scripts add. Crawlers that read only the server HTML, many AI crawlers among them, see what is shown below and nothing more.
The head
- Meta description
- lang
- Viewport
- hreflang
- Scripts
- Structured data
- None found in JSON-LD, microdata or RDFa.
No headings in the server HTML.
The text a crawler reads
No body text in the server HTML.
- internal
- external
- not followed
- unique URLs
Show the links
No <a href> links in the server HTML, so a crawler finds no other pages from here.
- with alt text
- decorative, alt=""
- no alt attribute
That was one page. See every page the way a crawler does.
Crawl Cove reads every page it crawls from its HTML and runs the checks that match this view: noindex (robots-noindex), robots.txt rules (robots-blocks-all-crawlers), thin pages (thin-content) and images with no alt attribute (image-alt). On the pages its rendered pass samples (10% of each crawl on a new installation, using Google Chrome or Microsoft Edge), it also compares raw and rendered pages (content-depends-on-js).
Download Crawl Cove freeNote
At the top is the crawl and index verdict: the robots.txt line that applies to Googlebot on this URL, the meta robots and meta googlebot tags, the X-Robots-Tag header and the canonical. Below it is the page itself, stripped to what a crawler reads: the title, meta description, language, hreflang and structured data types, the H1 to H6 outline, the body text with a word count, every link marked internal or external and followed or not, and the images with and without alt text. When the HTML is an empty app shell or holds little text beside its scripts, it says so, because a crawler that does not run JavaScript sees only what is shown here. No sign-up and no email.
Use this tool often? Bookmark this page, so it is one click away next time.
Email me this result
Want a copy for your records or a client? Leave your email and we'll send you this result. It is written by a real person at Crawl Cove, not an automated blast, so it may take a little while to arrive. We'll only use your address to send it, unless you also tick the box below. Privacy policy.
Got it. We'll email your result to you. Nothing else lands in your inbox unless you ticked the tips box.
How to view a page as a search engine spider
-
Paste the URL
Enter any public address, with or without https://. Redirects are followed hop by hop, and the page you end up on is the one shown.
-
Choose who fetches it
Googlebot Smartphone is the default, since Google indexes the mobile version of a page. Switch to a desktop browser to see whether the site sends crawlers something different.
-
Read the verdict first
Whether robots.txt lets Googlebot crawl the URL, and whether a robots tag, header or canonical keeps it out of the index. If the page cannot be crawled, the rest of the view is what Google would have seen, not what it sees.
-
Read the page as text
Check the headings make an outline, the main copy is in the body text, the links you need are real links, and the JavaScript note is clear. Copy the text or share the result link.
What this spider simulator shows
The fetch: the final URL, every redirect hop with its status code, the content type, the size of the HTML and how long the server took to start answering. The directives: the robots.txt group and rule that apply to Googlebot on this path, each meta robots, meta googlebot and X-Robots-Tag directive with where it came from, the canonical, and a verdict on whether Google may index the page. The head: the title, the meta description, the lang attribute, how many hreflang alternates the page declares and the structured data types in its JSON-LD, microdata or RDFa. Then the page as a crawler reads it: the H1 to H6 outline in source order, the body text with scripts, styles and noscript blocks left out, the word count, the links with internal, external and nofollow counts, and the images, split into those with alt text, those marked decorative with alt="" and those with no alt attribute at all.
check the H1 and heading order · check every link for nofollow
How Googlebot reads a page
Google describes three stages: crawling, rendering and indexing. Googlebot first fetches the URL and reads the HTML the server returns. It checks robots.txt before it fetches, and it finds new URLs from links written as <a> elements with an href. Pages that return a 200 are then queued for rendering, where a headless Chromium runs the JavaScript, and the rendered page is what gets indexed. The queue means rendering can happen some time after the crawl, so anything that only exists after a script runs depends on that second step working. This tool shows the first stage: the HTML as fetched, which is also all that a crawler that never renders will ever see.
Crawling and indexing are two different questions
robots.txt controls crawling, not indexing. A Disallow rule stops Googlebot fetching the page, but the bare URL can still appear in results if other pages link to it, and because Google cannot fetch the page it never sees a noindex on it either. A noindex in the meta robots tag or the X-Robots-Tag header controls indexing, and Google can only obey it on a page it is allowed to crawl. The canonical is a third signal: it asks Google to index another URL in place of this one. The simulator reads all three and shows them together, so you can tell a page that is blocked from one that is crawled and then dropped.
check whether a page can be indexed · Indexed, though blocked by robots.txt
When the content needs JavaScript
Some sites send a near empty HTML page, often a single <div id="root"> or <div id="__next">, and build the content in the browser. Google can usually index those pages after rendering, but many other crawlers read the server HTML only, most AI crawlers included, so they see the shell. The simulator flags an empty app root, a body with little text beside several scripts, a thin page whose noscript block asks for JavaScript, and a page with no links in its HTML. Each warning states what is in the HTML, not a guess about the rendered page. Server-side rendering or pre-rendering the main copy, headings and links fixes most of it. A framework is not a problem on its own: a site built with React or Vue that renders on the server sends its full text in the HTML and gets no warning.
Content that depends on JavaScript · check AI crawler access in robots.txt
Googlebot user agents and what a site sends them
Googlebot crawls most sites as Googlebot Smartphone, because Google indexes the mobile version of a page, and the simulator sends that user agent unless you pick a desktop browser. A site can answer a crawler differently from a visitor, on purpose or through a plugin, a firewall or a cache, so comparing the two is a quick test. A request claiming to be Googlebot from our server is not from Google, and some sites check that and turn it away; when that happens the error says so and the browser option is the way round it. Google treats showing crawlers different content from visitors as cloaking.
Frequently asked questions
- What is a spider simulator?
- A tool that fetches a web page the way a search engine crawler does and shows you what it can read: the text, the links, the headings and the tags in the HTML, without the layout, images or scripts a browser adds. It answers what a bot sees when it visits the page.
- Is this spider simulator free?
- Yes. It is free, needs no sign-up and no email, and the only limit is a rate limit that stops one visitor sending hundreds of checks a minute.
- Does it show the page exactly as Google sees it?
- It shows the first thing Google sees: the HTML the server sends to a request with the Googlebot Smartphone user agent, before JavaScript runs. Google renders JavaScript later and indexes the rendered page, so on a site that builds its content in the browser Google may see more than this. For the rendered version of your own page, use the URL Inspection live test in Search Console.
- Why is the text on my page missing?
- Usually because a script adds it after the page loads, so it is not in the server HTML. The JavaScript note says when that looks likely. Other causes are text inside an iframe, which belongs to the framed page, and text drawn into an image or a canvas.
- Does robots.txt stop a page being indexed?
- No. robots.txt stops crawling. A blocked URL can still be indexed without its content if other pages link to it. To keep a page out of results, let it be crawled and add a noindex meta tag or X-Robots-Tag header.
- Can search engines see hidden text?
- Text hidden with CSS is still in the HTML, so a crawler reads it and it appears in the body text here. Whether Google gives it weight is another matter; Google treats text hidden to mislead searchers as spam, while content in tabs or accordions that a visitor can open is normal.
- Do AI crawlers run JavaScript?
- Many do not, and read the server HTML only, which is what this view shows. If your main copy, headings or links are missing here, those crawlers are likely missing them too.
- Why does the simulator only count some links?
- It counts <a> elements with an href that point to a web page, because those are the links Google follows. Same-page anchors, email and phone links, and click handlers that only work with JavaScript are not counted. The list shows the first 500 links; the counts cover all of them.
- Can I check every page on my site this way?
- Not here: this tool reads one URL at a time. Crawl Cove, the desktop crawler, reads every page it crawls from its HTML, flags thin and noindexed pages, and can render a sample to compare raw with rendered. It has a permanent Free plan.
- Can I share the result?
- Yes. After a check the page address includes the URL and user agent you chose, so copying it gives a link that runs the same check. Results are cached for ten minutes.
From one page to your whole site
This shows one page from its server HTML. Crawl Cove reads every page of a site the same way and can render a sample in a browser to compare, so a template that hides its content behind JavaScript shows up as a list of pages. See audit a JavaScript site, raw vs rendered or see how Crawl Cove crawls a whole site.
More free tools
More technical SEO tools →-
Website SEO Checker
Open tool → -
HTTP Header Checker
Open tool → -
Canonical Checker
Open tool →