Crawl Cove Bot: User Agent, robots.txt and How to Control It

Crawl Cove is a desktop SEO auditing app. Somebody installed it on their own computer, typed your address into it and pressed go, and the requests in your logs are that crawl reading your pages for broken links, redirects, missing titles and the like.

It is not a search engine. It does not index your site, add it to any database of ours, or republish, resell or train on your content. There is no central Crawl Cove crawler working through a list of websites: a crawl happens because a person started one, and usually that person is you, your developer or an agency you have hired.

The user agent

Every ordinary crawl request identifies itself like this:

CrawlCove-Crawler

There is no version or URL in it today, which is honest but not helpful, and we intend to change that. The app will keep accepting the old token, so rules you write now keep working.

Two things to know before you write a rule that names it. The string is configurable in the app, so a rule naming Crawl Cove only catches people who left the default alone. And the optional rendered pass, which reloads a sample of pages in a real browser so JavaScript runs, sends that browser's own user agent rather than ours. It is off by default.

Where the requests come from

From the machine of whoever is running the audit. No Crawl Cove server fetches your pages and nothing passes through us, so the source address is a home broadband line, an office connection or a VPN, like any visitor.

That is why we cannot publish an IP list: the addresses belong to our customers, they change constantly, and publishing them would mean publishing our users' home connections. For the same reason there is no reverse DNS record to check, as there is for Googlebot. The control that does not depend on identifying us is rate limiting.

What the crawler does, in order

1. Fetches /robots.txt before anything else, for the address it was given and for its www or apex twin, then applies those rules to every URL.
2. Reads any Sitemap: lines there and fetches those sitemaps, following index files two levels deep. It never guesses at /sitemap.xml: advertise no sitemap and none is requested.
3. Fetches the starting page, then the internal links it finds, and so on to the run's limits. The defaults are 500 pages and five levels deep.
4. Requests /llms.txt once, if robots.txt allows, to report whether you publish one.
5. On a site that already looks like WordPress from the HTML it has read, makes up to four more requests, to /wp-json/, /xmlrpc.php, /wp-content/uploads/ and /?p=1, still subject to robots.txt. They are GETs, never POSTs, so no pingback is triggered. Other sites get none of these.
6. Optionally reloads a sample of pages in a real browser so client-side JavaScript runs. Off unless the user switches it on.

Links to other sites are recorded but never followed, so a crawl of somebody else's site never reaches yours. Redirects are followed up to ten hops, and each request gives up after fifteen seconds.

robots.txt

robots.txt is loaded first and always obeyed. The app has no setting to ignore it, for any user, on any plan, not even on their own sites. A disallowed URL is kept out of the crawl queue rather than filtered from the report, so it is never requested at all, not even as the target of a redirect. If robots.txt is unreachable or returns a server error, the crawler treats the whole site as off limits and stops.

To refuse Crawl Cove everywhere:

User-agent: CrawlCove-Crawler
Disallow: /

To keep it out of particular areas and allow the rest:

User-agent: CrawlCove-Crawler
Disallow: /basket/
Disallow: /search
Disallow: /account/

The token must match ours exactly, ignoring case: a group headed User-agent: CrawlCove does not match. If no group names us, the User-agent: * group applies. That wildcard is the better rule if you would rather not maintain a list of crawler names, and it is the one that still holds when somebody has changed the user agent in the app.

Crawl rate and server load

By default the crawler keeps four requests in flight and starts a new one no more often than every 300 milliseconds: roughly three requests a second, capped at 500 pages. Users can change both, within limits of one to sixteen in flight and a gap of nothing up to sixty seconds.

The crawler does not read the Crawl-delay directive. We would rather say so than let you write a line that quietly does nothing. What it does listen to is your server:

• A 429 is retried at most twice, waiting the number of seconds your Retry-After header asks for, or backing off exponentially from half a second without one. Any single wait is capped at thirty seconds.
• One 429 anywhere in a run then slows the whole crawl to a single request at a time with a three second gap, about twenty a minute, for the rest of that run. It does not speed back up.
• 503, 502 and 504 are treated as temporary and retried on the same schedule, without that lasting slowdown. Other 4xx responses are taken at face value and not retried.

So the most effective way to protect a small server is to rate limit and answer with 429. The crawler acts on it immediately and for the whole run, and it costs you nothing if the audit is one you wanted.

Blocking at your server or firewall

If you want a hard block rather than a polite one, these match on the user agent. nginx:

if ($http_user_agent ~* "CrawlCove-Crawler") {
    return 403;
}

Apache, in .htaccess:

RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} CrawlCove-Crawler [NC]
RewriteRule .* - [F,L]

Cloudflare, as a WAF custom rule with the action set to Block:

(http.user_agent contains "CrawlCove-Crawler")

What happens to the data

Nothing crawled from your site reaches us. Pages, findings and audit history are written to a database file on the machine that ran the crawl, and the app contacts our servers only to check a licence and look for updates. We hold no copy of your site, so there is nothing for us to share, sell or hand over, and nothing for you to ask us to delete. The detail is on the security page and in the privacy policy.

Report a problem

If a Crawl Cove crawl caused you trouble, email support@crawlcove.com and a person will read it. We cannot stop a crawl running on somebody else's computer, but we can tell you what you are looking at, and a pattern that keeps causing trouble is a bug we want to fix. Please include:

• Timestamps from your logs, with the time zone.
• The user agent string exactly as it was recorded.
• The source IP addresses, and the URLs that were hit.
• What went wrong: the request rate, the errors your server returned, or paths you did not expect to see requested.

Questions site owners ask