Retrying Failed Requests with Tenacity

Transient failures are not an exception at scale, they are a constant background rate, and this page — part of Asynchronous Scraping with Asyncio and HTTPX — shows how tenacity turns them into a bounded, observable cost instead of a crash.

Exponential backoff with jitter widens the gap between retries and caps it; fixed-interval retries stay flat and hammer a struggling server.

tenacity wraps any callable in a declarative retry policy, so you stop hand-rolling loops and sleep calls. The durable recipe for scraping has four parts: exponential backoff so a struggling server gets more room on each attempt, jitter so concurrent workers do not retry in lockstep, a stop condition on both attempts and total elapsed time, and a predicate that retries transport errors and the transient status codes (429, 500, 502, 503, 504) while failing immediately on everything else. Install it with pip install tenacity — it has no dependencies beyond the standard library.

Why Naive Retries Make Things Worse

The instinctive fix, while True around a time.sleep(1), fails in both directions. A fixed short delay retries aggressively into a server that is already returning 503s, adding load exactly when the target has least capacity. A fixed long delay wastes wall-clock on failures that would have cleared in 50 ms. Neither adapts, and without a stop condition a permanently broken endpoint turns into an infinite loop that quietly consumes a worker slot forever.

Exponential backoff fixes the timing: 1 s, then 2 s, then 4 s, then 8 s. Each failure buys the server more recovery time, and a genuinely transient blip still resolves quickly because the first retry comes almost immediately.

That introduces the second problem. If a hundred workers hit the same 503 in the same second, pure exponential backoff makes them all wait the same 1 s, then the same 2 s, and so on — they stay synchronized and arrive as a coordinated spike, which is a good way to keep a recovering server down. Jitter breaks the synchronization by adding a random offset to each wait. tenacity's wait_exponential_jitter(initial=1, max=30, jitter=1) computes initial * 2 ** retry_number, adds a uniform random value between 0 and jitter seconds, and clamps the result at max. Note that the jitter parameter defaults to 1 second — it is an additive nudge, not a full randomization of the interval, so if you want wide spreading you need to raise it or use wait_random_exponential instead.

Which Failures Are Worth Retrying

A retry budget is a finite resource. Spending it on a request that will fail identically every time is pure latency, and on a scraper it also delays the useful work behind it in the queue.

Only transport errors and the transient status codes deserve another attempt. A 4xx you caused will return the same answer every time, so spending the retry budget on it is pure delay.

The dividing line is whether anything can change between attempts. A timeout, a connection reset or a DNS failure means the request may never have reached the application — retry it. A 429 means the server explicitly asked you to slow down and try later — retry it, after the delay it named. A 502 or 503 means an upstream or a whole instance is unhealthy — retry it, because the load balancer may pick a healthy one next time.

A 400 means your request was malformed, a 401 means your credentials are wrong, a 403 means you are blocked, and a 404 means the resource is not there. None of those change because you asked again. The exception worth calling out: a 403 from an anti-bot layer can clear if the retry goes out through a different exit IP, but that is a proxy-rotation decision made at a different layer, not a tenacity decision — see Rotating Proxies and Managing IP Blocks.

Building the Policy

tenacity composes a policy from three orthogonal pieces: stop decides when to give up, wait decides how long to pause, and retry decides which outcomes count as failures. Because HTTP error statuses are returned rather than raised by default, the pattern is to raise a small custom exception for retryable statuses so the predicate can match on it.

import logging

import httpx
from tenacity import (
    before_sleep_log,
    retry,
    retry_if_exception_type,
    stop_after_attempt,
    stop_after_delay,
    wait_exponential_jitter,
)

logging.basicConfig(level=logging.INFO)
logger = logging.getLogger("scraper")

HEADERS = {
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 "
    "(KHTML, like Gecko) Chrome/125.0 Safari/537.36",
    "Accept": "text/html,application/xhtml+xml",
}
RETRYABLE_STATUS = frozenset({429, 500, 502, 503, 504})


class RetryableStatusError(Exception):
    """Raised for HTTP statuses that are worth another attempt."""


@retry(
    stop=stop_after_attempt(5) | stop_after_delay(120),
    wait=wait_exponential_jitter(initial=1, max=30, jitter=2),
    retry=retry_if_exception_type((httpx.TransportError, RetryableStatusError)),
    before_sleep=before_sleep_log(logger, logging.WARNING),
    reraise=True,
)
def fetch(client: httpx.Client, url: str) -> httpx.Response:
    resp = client.get(url, headers=HEADERS, timeout=15.0)
    if resp.status_code in RETRYABLE_STATUS:
        raise RetryableStatusError(f"{resp.status_code} for {url}")
    resp.raise_for_status()
    return resp


if __name__ == "__main__":
    with httpx.Client(follow_redirects=True) as client:
        page = fetch(client, "https://books.toscrape.com/")
        print(page.status_code, len(page.text), fetch.statistics)

Four details in that decorator are worth naming explicitly:

stop_after_attempt(5) | stop_after_delay(120) — the | operator combines stop conditions, and either one firing ends the sequence. Attempt caps alone do not bound wall-clock, because each attempt can itself take 15 s.
wait_exponential_jitter(initial=1, max=30, jitter=2) — bounded growth with a two-second random spread.
reraise=True — without it, exhaustion raises tenacity.RetryError, and the original httpx.ConnectTimeout is buried in .last_attempt.exception(). With it, your except httpx.TransportError blocks upstream keep working.
before_sleep_log — emits a WARNING before every sleep. A rising retry rate is one of the earliest signals that a target has started throttling you, which is why it belongs in the signals covered by Monitoring and Alerting for Scrapers.

fetch.statistics is a useful debugging handle: tenacity attaches a dict with attempt_number, idle_for (total seconds slept) and start_time to the decorated function after each call.

What the Attempt Budget Actually Costs

It is worth knowing the wall-clock a policy commits you to, because a retry budget multiplies across a queue. A five-attempt policy with initial=1 spends roughly seventeen seconds of sleeping before it gives up, on top of the time the four failed requests themselves took.

With initial=1 the base wait doubles on every retry and jitter adds up to one extra second, so a five-attempt policy spends roughly seventeen seconds before it gives up.

Multiply that by concurrency to see the real cost: at 20 workers, a target that starts returning 503 across the board parks all twenty for the better part of a minute each. That is usually the correct behaviour — you want to back off a failing site — but it means the throughput of a crawl degrades gracefully rather than falling off a cliff, and your queue depth will grow. If a whole host is failing, a circuit breaker that stops sending requests entirely for a minute is a better tool than a per-request retry.

Async Retries and Honouring Retry-After

tenacity handles coroutines transparently: decorate an async def and it awaits an asyncio.sleep between attempts rather than blocking the loop, so other tasks keep running. This composes cleanly with the concurrency ceiling described in Limiting Concurrency with Semaphores — the semaphore slot stays held while a task sleeps, which is what you want, because a retrying task should not free capacity for yet another request to the same struggling host.

A polite scraper should also read Retry-After on a 429 rather than guessing. The header comes in two forms: a number of seconds (Retry-After: 30) or an HTTP date (Retry-After: Wed, 21 Oct 2026 07:28:00 GMT). Handle both, and clamp the result so a hostile or broken value cannot park a worker for an hour.

import asyncio
import email.utils
import datetime as dt

import httpx
from tenacity import (
    retry,
    retry_if_exception_type,
    stop_after_attempt,
    wait_exponential_jitter,
)

HEADERS = {
    "User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 "
    "(KHTML, like Gecko) Version/17.4 Safari/605.1.15",
    "Accept": "application/json",
}
MAX_RETRY_AFTER = 120.0


class RateLimited(Exception):
    """Raised on 429 so the retry policy takes over."""


def parse_retry_after(raw: str | None, default: float = 5.0) -> float:
    if not raw:
        return default
    try:
        return min(float(raw), MAX_RETRY_AFTER)
    except ValueError:
        pass
    parsed = email.utils.parsedate_to_datetime(raw)
    if parsed is None:
        return default
    delta = (parsed - dt.datetime.now(dt.timezone.utc)).total_seconds()
    return min(max(delta, 0.0), MAX_RETRY_AFTER)


@retry(
    stop=stop_after_attempt(4),
    wait=wait_exponential_jitter(initial=2, max=60, jitter=3),
    retry=retry_if_exception_type((httpx.TransportError, RateLimited)),
    reraise=True,
)
async def fetch(client: httpx.AsyncClient, url: str) -> httpx.Response:
    resp = await client.get(url, headers=HEADERS, timeout=15.0)
    if resp.status_code == 429:
        await asyncio.sleep(parse_retry_after(resp.headers.get("Retry-After")))
        raise RateLimited(f"429 for {url}")
    resp.raise_for_status()
    return resp


async def main() -> None:
    async with httpx.AsyncClient(http2=True) as client:
        resp = await fetch(client, "https://httpbin.org/get")
        print("ok", resp.status_code, resp.http_version)


if __name__ == "__main__":
    asyncio.run(main())

Sleeping for Retry-After and then letting the backoff policy add its own wait is deliberate: the server's number is a floor, not a ceiling, and doubling on repeated 429s is the behaviour you want from a client that keeps getting told to slow down.

Edge Cases and Caveats

Do not retry non-idempotent writes blindly. A POST that succeeded but timed out on the response will be replayed. If your task queue writes records, guard with an idempotency key or an ON CONFLICT upsert — see Saving Scraped Data to PostgreSQL.
stop_after_attempt(n) counts attempts, not retries. stop_after_attempt(5) makes at most five calls, so four retries. Off-by-one here silently changes your budget by 25%.
A retry does not reset the request timeout. Each attempt gets its own 15 s, so five attempts can block a worker for 75 s of request time plus 17 s of sleeping. Combine with stop_after_delay if that matters.
Retrying a rendered page is far more expensive. In a browser-based crawl each attempt re-launches a context, so cap attempts lower for anything driven through Using Playwright for Modern Web Automation.
retry_if_result is the alternative to a custom exception. If you would rather not raise, retry=retry_if_result(lambda r: r.status_code in RETRYABLE_STATUS) works, but then reraise has nothing to re-raise and the caller receives the last bad response instead of an error.
Scrapy already has this. If you are inside a Scrapy project, RETRY_TIMES and RetryMiddleware cover the same ground at the downloader layer; adding tenacity inside a callback duplicates it. See Web Scraping with Scrapy.
Log the give-up, not just the sleeps. before_sleep_log fires between attempts but says nothing when the policy finally exhausts. Catch the re-raised exception at the call site and record the URL, or failures vanish from your dataset silently.

Frequently Asked Questions

What is the difference between exponential backoff and jitter? Backoff grows the wait between attempts — 1 s, 2 s, 4 s, 8 s — so a struggling server gets progressively more room. Jitter adds a random offset to each of those waits so that many workers failing at the same instant do not retry at the same instant. You want both: backoff for adaptivity, jitter to stop a hundred workers recreating the spike that caused the failure.

Which HTTP status codes should I retry? Retry 429 and the transient server errors 500, 502, 503 and 504, plus transport-level exceptions such as timeouts, connection resets and DNS failures. Do not retry 400, 401, 403 or 404 — those describe a problem with the request itself, and a second identical request produces an identical answer while consuming your budget.

Does tenacity work with async functions? Yes. Decorating an async def with @retry makes tenacity await an asynchronous sleep between attempts, so the event loop stays free to run other tasks while one is backing off. The policy syntax is identical to the synchronous case, and the same decorator arguments apply.

How do I stop retries from looping forever? Always supply a stop condition, and prefer two. stop_after_attempt(n) bounds the number of calls and stop_after_delay(seconds) bounds total elapsed time; combine them with the | operator so whichever fires first ends the sequence. Without a stop condition a permanently broken endpoint retries indefinitely.

Asynchronous Scraping with Asyncio and HTTPX — the parent topic covering event loops, clients and async crawl structure.
httpx vs aiohttp Async Performance — choosing the client the retry policy wraps.
Detecting Silent Scraper Failures — what to watch when retries stop being enough.
How to Scrape a Static Website Without Getting Blocked — the politeness rules a retry policy enforces.