429 Too Many Requests: what it means and how to fix it

A 429 Too Many Requests error means the client sent more requests than the server allows in a period of time, so the server is refusing further requests for now. It is the standard rate-limiting response. If you are the client, wait and retry more slowly, honouring the Retry-After header when there is one. If you run the server, make sure your limits don't throttle real users or search engine crawlers.

429 is defined in RFC 6585: it "indicates that the user has sent too many requests in a given amount of time" (RFC 6585, section 4). The response should explain the condition and may include Retry-After to say how long to wait. The RFC deliberately leaves the counting to the server: it "does not define how the origin server identifies the user, nor how it counts requests". A limit can apply per resource, per server, per API key, per logged-in user or per IP address.

What a 429 response looks like

The RFC's own example:

HTTP/1.1 429 Too Many Requests
Content-Type: text/html
Retry-After: 3600

<html>
  <head><title>Too Many Requests</title></head>
  <body>
    <h1>Too Many Requests</h1>
    <p>I only allow 50 requests per hour to this Web site per logged in user. Try again soon.</p>
  </body>
</html>

Retry-After takes either a number of seconds, as here, or an HTTP date such as Fri, 31 Dec 1999 23:59:59 GMT (RFC 9110, section 10.2.3). The RFC also says 429 responses "MUST NOT be stored by a cache", so a CDN should never serve a cached 429 to the next visitor.

Fixing a 429 when you are the client

You hit someone else's limit, whether an API, a scraper target or a SaaS dashboard. The fix is to send fewer requests, more slowly:

1. Honour Retry-After. It is the server telling you exactly when to come back.
2. Back off exponentially when there is no header. Double the wait after each 429 and add random jitter so parallel workers don't retry in lockstep.
3. Cap your concurrency. Ten parallel workers hitting a limit of five requests per second will keep failing however good the retry logic is.
4. Cache and batch. Don't re-fetch data that hasn't changed, and use batch endpoints where the API has them.
5. Check which limit you hit. Many APIs return headers describing the limit and what is left of it. Read the provider's docs, since these header names are not part of the RFC.

A short Python retry loop that honours Retry-After and falls back to exponential backoff:

import random
import time
from email.utils import parsedate_to_datetime

import requests


def get_with_backoff(url, max_attempts=5):
    for attempt in range(max_attempts):
        response = requests.get(url, timeout=30)
        if response.status_code != 429:
            return response
        retry_after = response.headers.get("Retry-After")
        if retry_after and retry_after.isdigit():
            wait = int(retry_after)
        elif retry_after:
            wait = max(0, parsedate_to_datetime(retry_after).timestamp() - time.time())
        else:
            wait = 2 ** attempt + random.random()
        time.sleep(wait)
    raise RuntimeError(f"Still rate limited after {max_attempts} attempts: {url}")

Setting rate limits when you are the server

Rate limits protect your server, but a limit set too tight turns real traffic into errors:

• Return 429, not 403 or 503, when a client is over its limit. 429 tells well-behaved clients to slow down and come back.
• Send Retry-After so clients know when to return instead of guessing.
• Limit by the right key. Per IP is simple, but many users can share one IP behind an office network or mobile carrier. Per API key or per account is fairer where you have one.
• Exempt health checks and your own monitoring, or your uptime alerts will fire on your own rate limit.

The SEO risk: rate-limiting Googlebot and AI crawlers by accident

A CDN or firewall rate limit applies to every client unless you exclude some. Crawlers fetch many URLs in a short time, which is exactly the pattern a rate limit catches.

Google's crawlers "treat the 429 status code as a signal that the server is overloaded", and "it's considered a server error". Like 5xx errors, it makes Google's crawlers "temporarily slow down with crawling", and URLs that keep returning server errors are eventually dropped from the index (Google Search Central). Google's crawl-rate guide says that returning 429 is acceptable to slow Googlebot "for a short period of time". It warns that if Googlebot sees these codes "on the same URL for multiple days, the URL may be dropped from Google's index" (Google Search Central).

To avoid throttling crawlers you want:

1. Check your logs for 429s served to crawler user agents such as Googlebot, Bingbot, GPTBot, OAI-SearchBot, ClaudeBot and PerplexityBot.
2. Verify crawlers before exempting them. User agents are easy to fake. Google recommends a reverse DNS lookup on the IP, then a forward lookup to confirm it, or matching against its published IP range files (Google Search Central). CDNs such as Cloudflare keep a verified bots list that does this check for you.
3. Exempt verified crawlers from aggressive limits, or give them a higher one, rather than switching the rate limit off for everyone.
4. Check robots.txt as well. A crawler can also be shut out by robots.txt rules. The AI crawler checker shows which AI and search crawlers your robots.txt allows or blocks. It reads robots.txt only, so a firewall rate limit won't show up there, which is why the log check comes first.

If you genuinely need Googlebot to slow down, for example during an overload, a short burst of 429 or 503 is Google's documented emergency brake. Remove it as soon as the server recovers.

429 vs 503

Both mean "not now". The difference is whose fault it is:

Use 429 when one client is over its limit, and 503 when the whole server is struggling or down for maintenance.