How to Fix Crawl Errors: Find, Sort and Validate in Search Console

To fix crawl errors, find the affected URLs in Search Console's Page indexing and Crawl stats reports, group them by cause, fix each cause at the server, CMS or link level, and then click Validate fix so Google rechecks them, as Google's Page indexing help describes.

Much of the work is sorting: a 404 on a page you removed on purpose needs no fix, while server errors on pages you want indexed need one quickly, because Google slows crawling when it sees them, per its HTTP status code documentation.

This guide walks through each error type in the order worth tackling, with the fix for each.

Key takeaways

• To fix crawl errors, group affected URLs in Search Console by cause, fix each cause at its source, then click Validate fix, as Google's Page indexing help describes.
• Google's crawlers slow down on 5xx or 429 responses and eventually drop URLs that keep returning server errors from the index, per its HTTP status code documentation.
• A 404 or 410 is the correct response for a removed page, and Google treats both the same way, per its HTTP status code documentation.
• After you click Validate fix, Google checks a sample of pages first, and validation typically takes up to about two weeks, according to Google's report help.

What counts as a crawl error?

A crawl error is any request where Googlebot could not fetch a URL or got an error back. Errors come at two levels. Site-level problems, such as robots.txt fetch failures, DNS failures and server connectivity problems, show up as host status in the Crawl stats report, according to Google's Crawl stats help. URL-level problems, such as server errors, 404s, soft 404s, redirect errors and blocked pages, show up as reasons in the Page indexing report.

Not every item in those reports is a problem. A 404 for a page that no longer exists is the correct response, and Google says sites under about 1,000 pages generally do not need the Crawl stats report at all, per the same help page.

Where to find crawl errors

• The Page indexing report lists each reason pages aren't indexed, with example URLs, in Google's report help.
• The Crawl stats report, under Settings, shows crawl requests by response and host status over the past 90 days for root-level properties, per its help page.
• The URL Inspection tool runs a live test on one URL and shows whether the page fetch succeeded, whether crawling was allowed and when Google last crawled it, per the URL Inspection help.
• Server logs show every Googlebot request, including the ones Search Console only samples. Confirm a request really came from Google with the reverse DNS check in Google's Googlebot verification guide.

Crawl errors by type

How to fix server errors (5xx)

Start here. Google's crawlers slow down when they get 5xx or 429 responses, URLs that keep returning server errors are eventually dropped from the index, and the crawl rate recovers gradually once the server answers with 2xx again, according to Google's HTTP status code documentation.

Work through the likely causes in order:

• Capacity. Compare error timestamps in your logs with traffic spikes, deploys and scheduled jobs. A server that falls over under a crawl burst also falls over for visitors.
• Timeouts. Slow database queries or uncached pages can run past a server or proxy timeout and turn into server errors.
• Bot protection. A CDN, firewall or security plugin that rate-limits Googlebot produces the same errors. Allow verified Googlebot traffic using Google's verification method rather than trusting the user agent, which anyone can fake.

How to fix robots.txt problems

Two different robots.txt problems show up in reports. The first is a rule that blocks pages you want crawled. Open the robots.txt report in Search Console, find the rule, fix it, and use Request a recrawl, as the robots.txt report help describes. Blocked URLs can still be indexed without their content, so robots.txt is not a way to remove pages; Google's technical requirements point to noindex for that.

The second is the file itself failing to load. Google's robots.txt documentation spells out what happens: a 4xx response other than 429 is treated as if no robots.txt exists, while a server error makes Google stop crawling the site for the first 12 hours, then use the last cached copy for up to 30 days, after which it behaves as if there is no file if the site is otherwise available. The same page says Google generally caches robots.txt for up to 24 hours and ignores anything past 500 KiB. Make sure the file returns 200 quickly on every host and protocol you serve.

How to fix 404 and soft 404 errors

A 404 is the right answer for a page that is gone. Google treats 404 and 410 the same way, removes previously indexed URLs that return them, and crawls them less often over time, per its HTTP status code documentation. So the fix is usually somewhere other than the missing URL:

• Find the pages that link to it, using your crawler's list of referring pages, and update or remove those internal links.
• Redirect the URL only when a page with substantially the same content exists. Otherwise leave it alone.
• Remove deleted URLs from your sitemap.

A soft 404 is the opposite problem: the page tells visitors it is missing, or is empty, but the server returns 200. Google flags these in the Page indexing report, and its crawl budget guide notes that soft 404 pages keep getting crawled and waste budget. Fix them by returning a real 404 or 410 for missing content, or by adding what a thin page lacks.

Single-page apps produce soft 404s easily, because the client-side router renders a "not found" view while the server returns 200. Google's JavaScript SEO basics suggests redirecting to a URL that returns a server-side 404, or adding a noindex tag with JavaScript. Our post on how JavaScript frameworks affect SEO covers the framework-level fixes.

How to fix redirect errors

The Redirect error reason covers chains that are too long, loops and bad redirect URLs, per Google's report help. Googlebot follows up to 10 redirect hops, but Google advises redirecting straight to the final destination, or keeping chains ideally to no more than 3 and fewer than 5, in its site move guidance. Map each old URL to its final target and update internal links so they skip the redirect entirely.

XeoRank's free redirect checker follows a URL hop by hop across its http, https and www variants, so loops and long chains are easy to see.

How to fix 401 and 403 blocks

Googlebot does not log in, so pages behind authentication come back as URL blocked due to unauthorized request (401), and a server that refuses Googlebot shows up as URL blocked due to access forbidden (403), per Google's report help. If the page is meant to be private, nothing needs fixing. If it is public, check firewall rules, CDN bot settings, geo-blocking and security plugins. Google's robots.txt documentation also warns against using 401 or 403 responses to limit crawl rate.

How to fix DNS and connectivity problems

If the Crawl stats host status shows DNS resolution or server connectivity issues, the problem sits below your site's code. Google's Crawl stats help gives an example: DNS failing for more than 5% of requests on a given day counts as an issue. Check that your DNS provider answers for every hostname you use, that records did not change during a migration, and that your host is not dropping connections under load.

How to fix sitemap errors

The Sitemaps report in Search Console shows whether Google could read each sitemap and lists processing errors. Typical causes include relative URLs, files over the size limit, and listed URLs that redirect or return errors. Google requires absolute URLs and caps each file at 50,000 URLs or 50MB uncompressed, per its sitemap build guide. You can review a sitemap with the free sitemap checker before resubmitting it.

How to validate the fix

Once every affected URL is fixed, open the issue in the Page indexing report and click Validate fix. Google checks a sample of pages first, then the remaining affected URLs, and validation typically takes up to about two weeks, according to Google's report help. Fix every instance before you start, because validation stops when Google finds one that is still broken.

For a single important URL, the Request indexing button in URL Inspection asks Google to recrawl it. There is a daily limit on these requests, so Google's URL Inspection help suggests submitting a sitemap when many pages have changed.

Discovered or crawled, but not indexed

Two statuses look like crawl errors but are not. Discovered - currently not indexed means Google knows the URL but has not crawled it yet. Crawled - currently not indexed means Google crawled it and has not indexed it, and resubmitting it is not needed, per Google's report help.

Google's crawl budget guide is written for sites with roughly 1 million or more pages that change weekly, 10,000 or more that change daily, or a large share of URLs stuck in Discovered - currently not indexed. If your site is smaller, improve the pages and the internal links that point to them instead. The technical SEO checklist for developers covers those related checks.

The error list in Search Console can look long, but it is finite. Sort it by cause, fix server and robots.txt problems first, clean up links and redirects next, then validate and let Google confirm the result.