# Blocked Due to Access Forbidden (403) and 401

Source: https://crawlcove.com/blog/blocked-due-to-401-or-403
Published 2026-10-06 · Updated 2026-11-01 · By Lewis Jones

> Blocked due to access forbidden (403) in Search Console: what 401 and 403 mean, when a refusal is correct, and how to find the rule shutting Googlebot out.

## Key takeaways
- Both statuses mean the server refused Googlebot: 401 asks for credentials it cannot give, 403 refuses it outright
- For pages behind a login, both are the correct response and neither needs fixing
- For public pages, a 403 almost always means a firewall, bot-protection rule, geo-block or hosting setting is treating Googlebot as an attacker
- Google's documentation says Googlebot never provides credentials, so a 403 on a public page is the server answering a question nobody asked
- A crawl that is refused by the same rules shows which pages and sections are affected, and whether the refusal is bot-specific

"Blocked due to access forbidden (403)" and "Blocked due to unauthorized request (401)" are two Page indexing statuses with the same shape: Googlebot asked for a page and the server refused. They differ in why, and that difference tells you where to look. Both are the right response for a page that should be private. Both are a serious problem for a page that should be public, because unlike most statuses in the report they mean Google could not read the page at all. This post covers what each code means, when each is correct, and how to find the pages and the rules behind them.

## What "Blocked due to access forbidden (403)" and 401 mean

Google's definitions, from its [Page indexing report documentation](https://support.google.com/webmasters/answer/7440203):

For 401: "The page was blocked to Googlebot by a request for authorization."

For 403: "HTTP 403 means that the user agent provided credentials, but was not granted access. However, Googlebot never provides credentials, so your server is returning this error incorrectly."

A 401 is the server saying "log in first". Googlebot has no login, so it stops. A 403 is the server saying "no", with or without a login. Google's phrasing on the 403 is pointed: since Googlebot never sends credentials, a 403 on a page meant to be public is a server misconfiguration, not a page problem.

**Note:**
This post is one of a series on the Page indexing report. [Search console indexing issues](https://crawlcove.com/blog/search-console-indexing-issues) covers every status and which ones to act on.

## When the statuses are correct

Pages that should be private should return one of these codes, and either status is confirmation that they do:

- Member areas, customer dashboards and account pages
- Staging and preview environments protected by HTTP authentication
- Internal tools and admin panels
- Paid content behind a paywall that is not using Google's paywalled-content markup
- Download endpoints that require a licence or a session

A 401 or 403 is a better way to keep these out of search than robots.txt or noindex, because it does not rely on a crawler respecting a hint. Google cannot read the page, cannot index its content, and reports it here. Nothing to do.

If a private page is also linked widely, Google may still list the bare URL, as it does for [indexed though blocked by robots.txt](https://crawlcove.com/blog/indexed-though-blocked-by-robots-txt). For most private areas that does not matter. If it does, the page needs to be crawlable with a noindex, which is a trade-off worth thinking about before making it.

## When they are a real problem

A public page returning 401 or 403 to Googlebot has one of a short list of causes, and almost all of them are rules that treat Googlebot as something it is not.

- **Bot protection.** Web application firewalls, DDoS mitigation and bot managers block requests that look automated. Googlebot's user-agent, request rate and lack of a browser fingerprint look automated. A rule set to "challenge" or "block" suspicious traffic without an allow list for verified search engine crawlers returns 403 to Google. This is the single most common cause.
- **Geo-blocking.** A site that refuses requests from outside its home country refuses Googlebot, which crawls mostly from the United States. The site is invisible to Google and the owner cannot see why, because it works from their desk.
- **Rate limiting with the wrong code.** Some servers answer excess requests with 403 rather than 429. Googlebot's crawl bursts trip the limit and get refused.
- **Hosting-level restrictions.** Shared hosts and some managed platforms add IP or user-agent blocks at the server level, outside anything in the site's own code. A hosting "security" feature switched on by default is a frequent source.
- **Hotlink or referrer protection applied too broadly.** Rules meant to stop other sites embedding your images that end up refusing any request without a referrer, which is every crawler request.
- **A login wall that catches everything.** A site-wide authentication rule intended for one directory, or a maintenance-mode plugin that protects the whole site with a password and returns 401 to anyone without it.
- **Leftover staging protection.** HTTP basic auth left on after launch returns 401 to every visitor who is not the developer, and Google reports the entire site under the 401 status.

The quickest tell is scale. One page under 403 is usually a page-level setting. Every page under 403 or 401 is a server, firewall or CDN rule.

## How to diagnose it

1. **Run a live test in URL Inspection.** If it reports the page fetch as failed with a 401 or 403, the block is current and applies to Googlebot now. If the live test passes, the rule has changed or is intermittent.
2. **Request the page without a browser.** From a command line, with no cookies, with a bot-like user-agent string, from a different network. If that returns 403 and a browser returns 200, a rule is matching on something about the request rather than the page.
3. **Check the firewall and CDN logs.** Look for refused requests from Googlebot's user-agent around the last crawl time, and read which rule fired.
4. **Check the hosting panel.** Look for IP blocking, country blocking, bot blocking or "protect against scrapers" features.
5. **Check for HTTP authentication.** A 401 with a `WWW-Authenticate` header on a public page is basic auth left on.

## How to fix it

The fix is to let verified search engine crawlers through without weakening protection against everything else.

- **Allow verified Googlebot, not the user-agent string.** Anyone can send a request claiming to be Googlebot. Google publishes the IP ranges its crawlers use and supports reverse DNS verification; most firewalls and CDNs have a verified-bots list built on exactly that. Use it.
- **Use 429 for rate limiting, not 403.** A 429 tells Googlebot to slow down and come back; a 403 tells it the page is off limits.
- **Scope geo-blocks to what needs them.** If some content must be region-locked, lock that content, not the whole site. Googlebot crawls from the US, so a US block is a Google block.
- **Remove staging authentication after launch.** Put it on a checklist. It is the most embarrassing version of this status and one of the most common.
- **Match the directory.** A login rule written for `/members/` that also matches `/members-guide/` is a one-character fix.

After the fix, run the live test again, then request indexing. The status clears per URL as Google recrawls, and a site-wide block that has been lifted can take a while to work through; [how long Google takes to recrawl a page](https://crawlcove.com/lessons/how-long-does-google-take-to-recrawl) covers what to expect.

**Warning:**
Do not fix a 403 by allowing anything that claims to be Googlebot. The user-agent string is the first thing a scraper fakes, and an allow list based on it opens the door to everything you were trying to keep out. Verified-bot allow lists exist because the string alone proves nothing.

## What the statuses are not

A 404 means the server worked and the page is absent, which [not found 404 in Search Console](https://crawlcove.com/blog/not-found-404) covers. A 5xx means the server failed; see [server error (5xx) in Search Console](https://crawlcove.com/blog/server-error-5xx). And a robots.txt block means Google never asked. The 401 and 403 statuses sit between those: the server worked, the page exists, and the server chose to refuse.

## Finding refused pages with a crawl

Search Console reports the pages Googlebot was refused, days after the fact, with no indication of which rule did it. A crawler of your own hits the same rules, and that is useful: a refusal that applies to any automated client is a refusal that applies to Google.

[Crawl Cove](https://crawlcove.com/features) identifies itself with its own user-agent, and when a site refuses that with a 400, 403 or 406 at the seed URL it switches to a browser user-agent and tries again, so a block that matches on the user-agent string alone is both detected and worked around. A 401 is deliberately not retried with a different user-agent, because a login wall is not a user-agent problem, and the crawl reports it as what it is. Across the crawl, the `status-code` check reports every page that returned a 401 or 403 at Critical severity (404 and 410 have their own checks, at High and Medium), so a login rule written one directory too wide shows up as a block of refused pages under one path. On a site where the crawler was refused under every user-agent it tried, the crawl records that the origin is bot-protected, which is the same answer to the question "is this site refusing crawlers?" that Google is getting.

The [Live Index Status](https://crawlcove.com/guides/live-index-status) panel calls Google's URL Inspection API for any page and reports the page fetch state alongside the coverage state and last crawl date, so you can confirm Google is let through after the fix. Run a full [SEO audit](https://crawlcove.com/seo-audit) to see every refused page on the site in one list.

## Wrap-up

"Blocked due to unauthorized request (401)" and "Blocked due to access forbidden (403)" are the server saying no to Google. For private pages that is the point and nothing needs doing. For public pages it is a firewall, geo-block, rate limiter or leftover staging password treating Googlebot as an intruder, and Google cannot read the page until it stops. Verify Googlebot properly and let it through, use the right code for rate limiting, and scope every protection rule to the pages that need it.

## FAQ

### What does "Blocked due to access forbidden (403)" mean?
Google's definition is that HTTP 403 means the user agent provided credentials but was not granted access, and since Googlebot never provides credentials, the server is returning this error incorrectly. Google asked for the page as an anonymous visitor and the server refused.

### What does "Blocked due to unauthorized request (401)" mean?
Google's definition is that the page was blocked to Googlebot by a request for authorisation. The server asked for a login, which Googlebot cannot supply, so Google could not fetch the page.

### Are 401 and 403 errors bad for SEO?
Only on pages you want indexed. For a members area, a staging site or an admin panel, both codes are the right answer and the status is confirmation. For a public page, both mean Google cannot read it at all, which is as bad as the status gets.

### Why does Google get a 403 when the page loads fine for me?
Because the rule refusing Googlebot is looking at something you do not share with it: the user-agent string, the IP range, the request rate, the absence of cookies or a JavaScript challenge. Firewalls, bot managers and geo-blocks are the usual sources.

### How do I let Googlebot through a firewall?
Verify Googlebot by reverse DNS or Google's published IP ranges and allow those requests, rather than matching on the user-agent string alone, which anyone can fake. Most CDNs and bot managers have a verified-bots allow list for exactly this.