Key takeaways
- Screaming Frog's free version has a hard 500 URL crawl limit; a licence replaces it with a 5 million URL limit, not an unlimited one
- There's no official trick to lift the free-tier cap, but four native settings under Configuration > Spider > Limits let you spend your 500 URLs on the pages that matter
- Limit by URL Path, crawl depth limits and excluding query strings all reduce wasted crawl budget without changing the 500-URL ceiling
- If you need a full crawl of a bigger site with no cap at all, either buy the £199/year licence or use a crawler that never had the limit
Screaming Frog's SEO Spider is free to use with no time limit, but the free version stops at 500 URLs per crawl. If you run a site with more pages than that, you'll hit the wall before the crawl finishes, often without warning until the count runs out. Here's exactly what counts against that limit, four native settings that make a single 500-URL crawl go further, and what actually removes the cap when you outgrow it.
What the 500 URL limit actually covers
Screaming Frog's own documentation is direct about this: "without a licence the SEO Spider is limited to crawling a maximum of 500 URLs each crawl." That figure resets each time you start a new crawl, but within one crawl it covers every URL the Spider requests, not just the HTML pages you'd count by hand. Images, CSS files, JavaScript, PDFs and any other resource the crawler fetches all count against the same 500, checked against Screaming Frog's own site, September 2026.
That matters in practice: two sites with the same number of pages can behave very differently against the limit. A text-heavy site with few images might get a genuinely useful 500-page crawl. A visually heavy site with a dozen images per page can burn through the same 500 URLs covering a much smaller slice of it.
What a licence changes, and what it doesn't
A Screaming Frog licence costs £199 per licence, per year (checked against screamingfrog.co.uk/seo-spider/pricing, September 2026), and it does remove the practical crawl limit, along with unlocking JavaScript rendering, scheduling, and the Search Console and Analytics integrations that are switched off in the free version.
It's worth being precise about what "removes the limit" means, though. According to Screaming Frog's own configuration documentation, a licence replaces the 500 URL crawl limit with a 5 million URL limit, not an unlimited one. For the overwhelming majority of sites that's the same thing in practice, since 5 million URLs is well beyond what most businesses will ever crawl in one pass. But if you've been told a licence makes Screaming Frog "unlimited," the more accurate description is "a much, much higher number."
Four native ways to make your 500 URLs count
None of these lift the 500-URL ceiling itself; there's no official way to do that without a licence. What they do is make sure the crawl spends its budget on pages you actually care about, rather than whatever order the Spider happens to discover them in. All four live under Configuration > Spider > Limits.
1. Limit by URL Path
If you only need to audit one section of a site, such as /blog/ or /products/, this setting targets the crawl directly at that path instead of starting from the homepage and hoping the section you want gets covered before the 500 runs out. It's the single most useful setting here if you already know which part of the site matters for this audit.
2. Limit Crawl Depth
This caps how many link-clicks away from your start page the Spider will follow. Without a depth limit, a large site with deep navigation can spend most of its 500 URLs several levels down before the crawler ever reaches some of your higher-priority pages. Setting a depth limit of two or three keeps the crawl broad and shallow, which is usually what you want for a first-pass technical audit.
3. Limit Number of Query Strings
Many sites generate large numbers of URL variants through filters, sorting options or tracking parameters, e.g. ?sort=price&filter=colour. Each of those is a distinct URL as far as the crawler is concerned, and on an e-commerce or listings site they can dominate a 500-URL crawl without adding a single new page of real content. Excluding URLs above a set number of query strings keeps the crawl focused on genuinely different pages.
4. Limit Max Folder Depth
Similar to crawl depth, but counts subdirectory levels in the URL itself rather than link clicks from the homepage. Useful on sites where the folder structure and the click-depth from the homepage don't line up, such as a large documentation site with deep nesting that's still only a click or two from the nav.
When the free version is enough, and when it isn't
If your site is comfortably under 500 URLs, or you're auditing one section of a larger site, the free version with these four settings applied will get you a genuinely useful crawl at no cost. It's a good way to try Screaming Frog properly before deciding whether the paid features are worth it for your workflow.
Where it stops being enough is a full-site audit of anything past a few hundred pages, or any workflow that needs the features the free version switches off entirely: saving and reopening a crawl, scheduling, JavaScript rendering, or pulling in Search Console and Analytics data alongside the crawl. At that point the honest choice is between the £199/year Screaming Frog licence, or a crawler that never had a URL cap to begin with.
That's the gap Crawl Cove is built for: no URL limit on any plan, Search Console in one click, Bing and Core Web Vitals with a free key each, and every finding comes with a plain-English fix rather than raw data to interpret yourself. See the full breakdown against Screaming Frog, compare the pricing directly, or start a 14-day trial with no card required.
Frequently asked questions
- What exactly counts towards Screaming Frog's 500 URL limit?
- Every URL the Spider actually requests during a crawl, checked against screamingfrog.co.uk's own documentation, September 2026. That includes HTML pages, but also images, CSS, JavaScript, PDFs and any other resource the crawler fetches and adds to a tab, since each one is a request against the same total. A page with a lot of embedded images can burn through the 500 faster than the page count alone suggests.
- Does buying a licence make Screaming Frog's crawl limit unlimited?
- No, and it's a common misconception. A licence raises the limit from 500 URLs to 5 million URLs per crawl, according to Screaming Frog's own configuration documentation, checked September 2026. That's high enough that it behaves as unlimited for almost every real site, but it is a raised ceiling, not the absence of one.
- Is there a way to crawl more than 500 URLs for free?
- Not officially. Screaming Frog's own FAQ states plainly that a licence is what removes the limit, and we found no documented free workaround that lifts the 500-URL ceiling itself. What you can do for free is make sure those 500 URLs are the ones you actually want, using the native Limits settings covered below.
- What's the difference between Limit Crawl Depth and Limit by URL Path?
- Limit Crawl Depth caps how many clicks away from your start page the Spider will follow links, so it stops it wandering deep into low-value sections. Limit by URL Path instead lets you target a specific folder or section directly, so all 500 URLs go to, say, /blog/ rather than being spent in whatever order the crawler happens to discover pages. Use URL Path limiting when you know which section you care about; use depth limiting when you want a broad but shallow view of the whole site.