Crawl Cove's crawler is polite and safe by default, and predictable on purpose. Knowing how it behaves helps you explain a result to a client ("why was that page flagged?") and tune a crawl with confidence. This page is the full behaviour reference.
robots.txt — loaded first, always respected
The crawler fetches robots.txt before anything else, and disallowed URLs
never enter the crawl. There's no override toggle; respecting robots is the
default and the rule.
Its fallbacks are deliberately safe:
- A
404or410(no robots file) → allow all. - A
5xx, or a parse/transport error → treat the origin as off-limits (disallow all) and log it. When the crawler can't read the rules, it assumes the strictest ones rather than charging ahead.
Politeness & throughput
A single queue runs at concurrency 4 with a 300 ms minimum gap between request starts. Both are configurable per crawl, but the defaults cap throughput at roughly 3.3 requests per second — fast enough to finish, gentle enough not to look like an attack on a client's server.
Retries & rate limiting
| Behaviour | Detail |
|---|---|
| Retried statuses | 429, 502, 503, 504 (transient) |
| Max retries | Up to 2 retries — 3 attempts total |
| Backoff | Exponential, 500 ms base, capped at 30 s |
Retry-After |
Honoured when the server sends it |
| Genuine 4xx | Not retried |
This is why a momentarily throttled page isn't wrongly flagged as a critical error — the crawler backs off and tries again before giving up, and it never hammers a server that asked it to wait.
Sitemaps & redirects
- Sitemaps declared in robots.txt are followed, including nested sitemap indexes up to 2 levels deep — a cap that stops looping or malicious indexes from running away. A broken sitemap is skipped, never fatal.
- Redirects are followed manually so the crawler can capture the full chain
(up to 10 hops). That captured chain is what powers the
redirect-chainandcanonical-to-redirectchecks in the audit-check reference.
User-agent auto-negotiation
The crawl starts with the user-agent CrawlCove-Crawler. If the seed URL blocks
it (a 403 or 406), the crawler switches to a browser user-agent and
continues. The working UA is then remembered per site, so future and
scheduled crawls don't re-probe — they go straight to the UA that worked last
time.
Render sampling (Tier-2)
The Tier-2 rendered pass is deterministic, not random — the same crawl always renders the same pages. It picks an evenly-spaced sample (sized by your render-% setting), plus every page that had no JSON-LD in its raw HTML.
Tip
That "plus every page lacking raw JSON-LD" rule is the clever part: schema that a plugin injects only after JavaScript runs is never missed, which avoids the false "no structured data found" finding on perfectly fine pages.
WordPress auto-detection
Detection happens in two stages, and a non-WordPress site is never poked:
- Passive (no extra requests): a page's generator meta tag, or
wp-content//wp-includes/appearing in a link or image, marks the site as WordPress. - Active (only if passive matched, and only if robots allows): four bounded
GETs probe
/wp-json/,/xmlrpc.php(never POSTed, to avoid triggering pingbacks),/wp-content/uploads/, and/?p=1.
Non-WordPress sites get zero WP probes and zero WP findings. The WordPress check pack simply doesn't fire when the site isn't WordPress.
Caching (so you're not re-hitting APIs)
| Data | Cache |
|---|---|
| CrUX field data | 24 h per (URL, form factor); a real "no data" answer is cached too. |
| Open PageRank | 24 h per domain. |
| GSC sync | Pulls a 90-day window; GSC data itself lags ~2 days. |
| Bing backlinks | Live per-sync; new/lost computed across syncs (nothing deleted). |
Note
Caching a genuine "no data" answer (for CrUX) is intentional. A low-traffic URL with no field record won't trigger a fresh API call every time you open it — the honest empty result is remembered for 24 hours.