Skip to content

Robots.txt Tester & Generator

Build a robots.txt from scratch or fetch your live one, edit the rules, and test any URL against any crawler — Googlebot, Bingbot, GPTBot and more — to see instantly whether it’s allowed or blocked, and which line decides it. Everything but the initial fetch runs in your browser.

robots.txt

Add a rule

Test URLs

Tip

Verdicts use the same longest-match rules Google uses: the most specific Allow/Disallow wins, and Allow beats Disallow on an equal-length tie. Edit the file on the left and every test re-evaluates instantly.

Note

Google retired its standalone robots.txt Tester. This runs the same matching logic in your browser — nothing you type or fetch is stored. robots.txt controls crawling, not indexing: a blocked URL can still be indexed if other pages link to it, so use noindex to keep pages out of results.

Email me this result

Want a copy for your records or a client? Leave your email and we'll send you this result — written by a real person at Crawl Cove, not an automated blast, so it may take a little while to arrive. We'll only use your address to send it, unless you also tick the box below. Privacy policy.

That email address doesn't look right — please check it and try again.

Something went wrong saving your address — your result above is unaffected; please try again.

Got it — we'll email your result to you. Nothing else lands in your inbox unless you ticked the tips box.

Generate and test robots.txt in one place

Build rules from scratch or fetch your live robots.txt, then test any URL against any crawler to see instantly whether it’s allowed or blocked — and the exact line that decides it. It’s a modern replacement for Google’s robots.txt Tester, which was retired from Search Console at the end of 2023.

How robots.txt rules are matched

A crawler obeys the most specific group for its user-agent, falling back to the wildcard “*” group. Within that group the longest-matching Allow or Disallow path wins, and Allow beats Disallow on an equal-length tie. This tester applies exactly those rules, so your result matches how Google and other crawlers actually behave.

Frequently asked questions

What is a robots.txt file?
A plain-text file at your domain root that tells crawlers which URLs they may request. It uses User-agent groups with Allow and Disallow rules, and can point to your sitemap.
Does robots.txt stop a page being indexed?
No — robots.txt controls crawling, not indexing. A URL you disallow can still be indexed if other pages link to it. To keep a page out of Google’s results, use a noindex meta tag or header instead.
Where does robots.txt go?
At the root of your domain, always reachable at https://yourdomain.com/robots.txt. Crawlers only read it from that exact location — a subfolder copy is ignored.
How do I block AI crawlers in robots.txt?
Add Disallow: / rules for the AI user-agents you want to keep out — GPTBot, ClaudeBot, Google-Extended and so on. Use the “Block all AI bots” preset here, then test the result before deploying.
Why did Google retire its robots.txt Tester?
Google removed the standalone tester from Search Console in December 2023. This tool runs the same longest-match logic in your browser, so you can keep testing rules without it.

From one page to your whole site

This builds and tests one file; Crawl Cove tests the outcome across your whole site — flagging every important page accidentally blocked by robots.txt or noindex, so a single stray Disallow never quietly deindexes a section. See the features or compare the plans.

Download Crawl Cove

More free tools