Distributed Crawling with Celery and Redis

A single process can only fetch so much, and this guide — part of Scaling Python Web Scrapers — covers what to build when one machine is no longer enough. The scope is a crawl spread over several workers coordinated by a broker: task design, queue routing, deduplication that works across processes, per-domain rate limits, and the backpressure that stops a producer from filling Redis until it evicts your queue.

The shift from a single event loop to a distributed queue changes the unit of work. In an async crawler the state — which URLs are pending, which have been seen, how many requests are in flight — lives in process memory and dies with the process. In a queue-based crawler that state lives in the broker, so a worker is disposable: it can crash, be replaced, or be joined by nine more, and the crawl continues. Celery is the mature Python implementation of that model, and Redis serves as its broker, result backend, deduplication set and rate limiter in a single service.

When to Use a Distributed Task Queue

Reach for Celery and Redis when a single machine is no longer enough — but not before, because the moving parts add real operational cost. The architecture pays for itself when:

• The URL frontier is huge or unbounded. You discover new links as you crawl and cannot hold the whole queue in one process, or the crawl runs for days and must survive deploys.
• You want horizontal scale. Adding capacity should mean starting another worker, on another machine, pointed at the same broker — with no code change and no re-sharding.
• Work must survive crashes. A queued task should still be there after a worker dies, and be retried automatically rather than silently lost.
• Different stages have different costs. Cheap HTML fetches and expensive browser renders can run on separate worker pools sized independently.
• Egress must be spread across hosts. Many machines in different networks means many source addresses, which changes what a target sees.

If your crawl fits comfortably on one box, a single async event loop is simpler and faster to reason about — see Asynchronous Scraping with asyncio and HTTPX. And if you mostly need link-following with built-in retries on one machine, Web Scraping with Scrapy already gives you a scheduler and concurrency without a separate broker.

The costs are worth stating plainly. You now have a stateful service to run, monitor and back up. Task arguments must be serialisable, which rules out passing parsed trees or open connections between stages. Debugging moves from a stack trace in one terminal to correlating logs across five workers. And at-least-once delivery means every task must be idempotent, because it will occasionally run twice.

Prerequisites

Python 3.10 or newer and a Redis server 6.0 or newer. Celery 5.4 requires Python 3.8+; the examples use modern typing syntax that needs 3.10.

pip install "celery[redis]==5.4.0" "httpx==0.27.2" "redis==5.1.1" "selectolax==0.3.21"

Run Redis locally with Docker while you develop:

docker run --rm -p 6379:6379 redis:7-alpine --maxmemory 512mb --maxmemory-policy noeviction

The noeviction policy is deliberate. Redis's default for a memory-capped instance is to start deleting keys, which for a broker means silently deleting queued tasks. With noeviction, a producer that overruns the limit gets an error it can handle instead of losing work invisibly. Verify the connection before writing any tasks:

redis-cli -h 127.0.0.1 -p 6379 ping

Step-by-Step: Building the Distributed Crawler

1. Configure the Celery App

A single Celery instance names the app, points at the Redis broker (where tasks are queued) and result backend (where return values are stored), and sets the defaults that make a crawl survive worker loss. Keep this in one importable module so both the producer and the workers load identical configuration.

# crawler/app.py
from celery import Celery

app = Celery(
    "crawler",
    broker="redis://127.0.0.1:6379/0",
    backend="redis://127.0.0.1:6379/1",
    include=["crawler.tasks"],
)

app.conf.update(
    task_acks_late=True,              # re-queue a task if the worker dies mid-run
    task_reject_on_worker_lost=True,
    worker_prefetch_multiplier=1,     # fair dispatch for long-running fetches
    task_default_retry_delay=5,
    task_time_limit=120,              # hard kill a stuck fetch after two minutes
    task_soft_time_limit=100,         # raise SoftTimeLimitExceeded first, so cleanup runs
    task_serializer="json",
    result_serializer="json",
    accept_content=["json"],
    result_expires=3600,
    worker_max_tasks_per_child=200,   # recycle workers to release leaked memory
    broker_transport_options={"visibility_timeout": 300},
)

Three of these settings matter more than the rest. task_acks_late=True means a task is acknowledged after it completes rather than when it is received, so a worker killed mid-fetch returns the task to the queue instead of dropping it — at the cost of the task possibly running twice, which is why idempotence is mandatory. worker_prefetch_multiplier=1 stops a single worker from reserving a hundred tasks it will take ten minutes to finish while other workers sit idle. And visibility_timeout must exceed your longest task, or Redis will hand the same task to a second worker while the first is still working on it.

Using database 0 for the broker and 1 for the backend keeps queued work and results in separate Redis logical databases, which makes them easy to inspect and flush independently.

2. Design the Fetch Task

A task is a normal Python function decorated with @app.task. Bind it with bind=True so it has access to self for retries, keep its arguments small and JSON-serialisable, and always send an explicit User-Agent — anonymous default clients are the first thing anti-bot systems flag.

# crawler/tasks.py
import httpx
from celery import Task

from .app import app

HEADERS = {
    "User-Agent": (
        "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 "
        "(KHTML, like Gecko) Chrome/127.0.0.0 Safari/537.36"
    ),
    "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
    "Accept-Language": "en-GB,en;q=0.9",
}

RETRYABLE = {429, 500, 502, 503, 504}


@app.task(bind=True, max_retries=4, acks_late=True)
def fetch_url(self: Task, url: str) -> dict[str, object]:
    try:
        with httpx.Client(timeout=15, follow_redirects=True, headers=HEADERS) as client:
            response = client.get(url)
        response.raise_for_status()
    except httpx.HTTPStatusError as exc:
        status = exc.response.status_code
        if status not in RETRYABLE:
            return {"url": url, "status": status, "ok": False}
        wait = int(exc.response.headers.get("Retry-After", 2 ** self.request.retries))
        raise self.retry(exc=exc, countdown=wait)
    except httpx.TransportError as exc:
        raise self.retry(exc=exc, countdown=2 ** self.request.retries)

    return {
        "url": url,
        "status": response.status_code,
        "length": len(response.text),
        "ok": True,
    }

The 2 ** self.request.retries countdown is exponential backoff: waits of 1, 2, 4, then 8 seconds before Celery gives up after four attempts. Returning a small dictionary rather than the page body is deliberate — every return value is serialised into Redis, and shipping megabytes of HTML through the result backend will exhaust memory long before the queue does.

Creating a new httpx.Client per task discards connection pooling. For high-volume fetching, hold a module-level client and let the worker process reuse it across tasks; the connection pool then persists for the life of the worker rather than the life of one task.

3. Fan Work Out Across Purpose-Built Queues

The producer enqueues tasks with apply_async. Because each call returns immediately, one process can dispatch thousands of URLs in a tight loop while workers pick them up in parallel across every machine subscribed to the broker.

Routing to named queues is what lets you size pools independently. A fetch of static HTML needs almost no memory and benefits from high concurrency; a headless browser render needs a gigabyte and should run two at a time.

# crawler/producer.py
from .tasks import fetch_url


def enqueue(urls: list[str], queue: str = "fetch") -> int:
    for url in urls:
        fetch_url.apply_async(args=[url], queue=queue, expires=3600)
    return len(urls)


if __name__ == "__main__":
    seed = [f"https://books.toscrape.com/catalogue/page-{n}.html" for n in range(1, 51)]
    print(f"queued {enqueue(seed)} URLs")

Start workers on any number of machines, all pointed at the same Redis:

celery -A crawler.app worker --queues fetch --concurrency 16 --loglevel info --hostname fetch-1@%h
celery -A crawler.app worker --queues render --concurrency 2 --loglevel info --hostname render-1@%h

The expires=3600 argument on apply_async matters more than it looks. Without it, a queue that backs up during an outage will eventually be drained by workers fetching URLs that were relevant six hours ago. With it, stale tasks are discarded rather than executed.

For I/O-bound fetching, the gevent pool gives far higher concurrency per worker than the default prefork pool, because each request costs a greenlet rather than a process: celery -A crawler.app worker --pool gevent --concurrency 200. Prefork remains the right choice when tasks use CPU or call libraries that are not greenlet-safe.

4. Deduplicate with a Shared Redis Set

In a distributed crawl the same URL is discovered from many pages, and each worker has its own memory. A Redis set is the natural shared store: SADD is atomic, so only the first worker to see a URL enqueues it. Redis returns 1 for a newly added member and 0 for a duplicate.

# crawler/dedup.py
import hashlib

import redis

_pool = redis.ConnectionPool(host="127.0.0.1", port=6379, db=2, max_connections=32)

SEEN_KEY = "crawl:seen"


def _fingerprint(url: str) -> str:
    return hashlib.blake2b(url.encode("utf-8"), digest_size=16).hexdigest()


def claim_urls(urls: list[str]) -> list[str]:
    """Return only the URLs this caller is the first to claim."""
    client = redis.Redis(connection_pool=_pool)
    fingerprints = [_fingerprint(url) for url in urls]
    with client.pipeline(transaction=False) as pipe:
        for fingerprint in fingerprints:
            pipe.sadd(SEEN_KEY, fingerprint)
        added = pipe.execute()
    return [url for url, is_new in zip(urls, added) if is_new == 1]

Two changes from the obvious implementation are worth the effort. Storing a 16-byte fingerprint instead of the full URL cuts memory by roughly 75% — at fifty million URLs that is the difference between about 4 GB and about 1 GB of Redis. And pipelining the SADD calls turns one round trip per URL into one round trip per batch, which at a thousand URLs is a 100× reduction in latency.

Wire it into discovery so only unseen links are enqueued:

# crawler/discover.py
from .dedup import claim_urls
from .tasks import fetch_url


def enqueue_links(links: list[str]) -> int:
    fresh = claim_urls(links)
    for url in fresh:
        fetch_url.apply_async(args=[url], queue="fetch")
    return len(fresh)

Set an expiry on the set (client.expire(SEEN_KEY, 86400)) if you want the frontier to reset daily rather than growing forever. When even fingerprints are too large to hold, a probabilistic structure trades a small false-positive rate for constant memory — see Deduplicating URLs with Bloom Filters.

5. Rate-Limit Per Domain, Not Per Task

Celery's built-in rate_limit throttles a task type globally, which is the wrong granularity: a crawl touching two hundred domains should not slow down on domain B because domain A is busy. A fixed-window counter in Redis gives per-domain limits cheaply and atomically across every worker.

# crawler/ratelimit.py
import time

import redis

_pool = redis.ConnectionPool(host="127.0.0.1", port=6379, db=2, max_connections=32)


def allow_request(domain: str, max_per_second: int = 4) -> bool:
    client = redis.Redis(connection_pool=_pool)
    bucket = f"rate:{domain}:{int(time.time())}"
    with client.pipeline(transaction=True) as pipe:
        pipe.incr(bucket)
        pipe.expire(bucket, 2)
        count, _ = pipe.execute()
    return count <= max_per_second

Inside the task, requeue politely instead of hammering when the budget is spent:

# crawler/tasks.py (inside fetch_url, before the request)
from urllib.parse import urlsplit

from .ratelimit import allow_request

domain = urlsplit(url).netloc
if not allow_request(domain):
    raise self.retry(countdown=1, max_retries=None)

A fixed window has a known weakness: two bursts either side of a second boundary can produce double the intended rate for a moment. For polite crawling that is acceptable. If it is not, a sliding-window sorted set (ZADD with a timestamp score, ZREMRANGEBYSCORE to trim, ZCARD to count) gives an exact limit at roughly three times the Redis cost per check.

6. Apply Backpressure to the Producer

A producer that enqueues faster than workers consume will fill Redis. Because the queue lives entirely in memory, the failure is not gradual: the instance hits maxmemory and either refuses writes or, under a default eviction policy, starts deleting queued tasks that nobody will ever notice were lost.

The fix is a producer that checks depth before dispatching:

# crawler/backpressure.py
import time

import redis

_pool = redis.ConnectionPool(host="127.0.0.1", port=6379, db=0, max_connections=8)

MAX_PENDING = 100_000


def wait_for_capacity(queue: str = "fetch", ceiling: int = MAX_PENDING) -> int:
    """Block until the queue has room, then return the current depth."""
    client = redis.Redis(connection_pool=_pool)
    while True:
        depth = client.llen(queue)
        if depth < ceiling:
            return depth
        time.sleep(2.0)


def enqueue_with_backpressure(urls: list[str], batch_size: int = 500) -> int:
    from .tasks import fetch_url

    sent = 0
    for start in range(0, len(urls), batch_size):
        wait_for_capacity()
        for url in urls[start : start + batch_size]:
            fetch_url.apply_async(args=[url], queue="fetch", expires=3600)
            sent += 1
    return sent

Celery stores a Redis-backed queue as a list named after the queue, so LLEN fetch is the pending count. Alerting on that number is the single most useful metric a distributed crawl has: a depth that trends monotonically upward means the fleet is undersized, and a depth that drops to zero while URLs remain undiscovered means the producer has stalled. Wiring it into a dashboard is covered in Monitoring and Alerting for Scrapers.

Performance and Scaling Considerations

• Match the pool type to the workload. For I/O-bound fetches, --pool gevent --concurrency 200 keeps hundreds of sockets busy in one process. For CPU-heavy parsing, keep the prefork pool with concurrency near the core count, and route parsing to its own queue so it cannot starve fetching.
• Use dedicated queues and size them separately. Cheap fetches on a fetch queue with many workers, expensive browser renders on a render queue with few. A single mixed queue forces every worker to be provisioned for the heaviest task.
• Run async inside each worker. A task can itself fetch a batch of twenty URLs with an event loop, combining process-level distribution with in-process concurrency. That reduces broker round trips by a factor of twenty, which matters once the broker is the bottleneck.
• Watch the broker as carefully as the workers. Redis holds the queue in memory. Cap the frontier, set maxmemory with noeviction, and alert on LLEN per queue and on used_memory from INFO memory.
• Keep the result backend small. Storing every return value in Redis adds up fast. Set result_expires, use ignore_result=True for fire-and-forget fetches, and persist the real data to a database from inside the task — see Storing and Exporting Scraped Data.
• Expect at-least-once delivery. With acks_late, a task that times out or whose worker dies will run again. Every task must therefore be safe to run twice: upsert on a natural key rather than insert, and make side effects idempotent.
• Deploy workers as containers behind a scheduler. Scaling then means changing a replica count, and a crashed worker is replaced automatically — the patterns are in Deploying Scrapers to the Cloud.

Common Errors and Fixes

kombu.exceptions.OperationalError: Error 111 connecting to 127.0.0.1:6379. Connection refused. The broker is unreachable. Confirm Redis is running (redis-cli ping returns PONG) and that the broker= URL host, port and database match. Inside Docker networks, use the service name rather than 127.0.0.1, which resolves to the container itself.

Tasks queue but never run. The worker is listening on a different queue than the producer targets. If you enqueue with queue="fetch", start the worker with --queues fetch. Without --queues, workers consume only the default celery queue, so the fetch list grows untouched.

Received unregistered task of type 'crawler.tasks.fetch_url'. The worker did not import the task module. Add include=["crawler.tasks"] to the Celery(...) constructor or call app.autodiscover_tasks(["crawler"]), and confirm the worker's working directory puts the package on sys.path.

WorkerLostError: Worker exited prematurely: signal 9 (SIGKILL). The OS killed a worker, usually out-of-memory from oversized responses or too-high concurrency. Lower --concurrency, stream or truncate large bodies, and set worker_max_tasks_per_child so workers recycle and release leaked memory.

billiard.exceptions.SoftTimeLimitExceeded. A task exceeded task_soft_time_limit. This is the good outcome — the exception is raised inside your task so a try/finally can close connections and flush partial results before the hard limit kills the process. Catch it explicitly around long fetches.

The same URL is fetched repeatedly despite the dedup set. Either the claim happens after apply_async instead of before, or two code paths enqueue — the producer and a discovery callback. Do the SADD exactly once, at the moment a URL is discovered, and enqueue only what it returns.

Queue depth grows while worker CPU sits near zero. The workers are blocked, not busy: usually on a per-domain rate limiter that keeps re-queueing, or on a downstream database that has stopped accepting connections. Check the retry counters before adding more workers, because more workers will make a contended lock worse.

redis.exceptions.ResponseError: OOM command not allowed when used memory > 'maxmemory'. The producer outran the fleet and Redis is full. This error is the correct behaviour under noeviction — it tells you the crawl is unbalanced instead of silently dropping tasks. Add backpressure to the producer and either scale workers up or lower the enqueue rate.

Frequently Asked Questions

Do I have to use Redis, or can Celery use RabbitMQ? Both are supported. RabbitMQ is a dedicated message broker with richer routing and stronger delivery guarantees; Redis is simpler to operate, faster to stand up, and doubles as your deduplication set, cache and rate limiter. For most scraping workloads Redis is the pragmatic choice. If Celery itself feels heavy, the trade-offs against a lighter framework are in Celery vs RQ for Scraping Task Queues.

How is this different from just running Scrapy? Scrapy distributes concurrency within one process and one machine extremely well, but its scheduler and frontier live in that process. Celery distributes tasks across machines and keeps the frontier in a durable broker that survives restarts. Many teams run Scrapy spiders inside Celery tasks to get both.

How do I stop workers from getting IP-banned at scale? Per-domain rate limiting is the first defence, but many workers usually share a small number of egress addresses, so the target sees one very busy client. Route requests through rotating proxies so requests arrive from many source addresses — see Rotating Proxies and Managing IP Blocks.

Where should scraped results actually be stored? Not in the Celery result backend beyond the short term. Return small metadata from tasks so the backend stays lean, and write the extracted records to a database or object store from within the task itself.

Can I schedule recurring crawls with Celery? Yes — Celery Beat is a built-in scheduler that enqueues tasks on a cron-like timetable. Run exactly one Beat process, or every schedule fires once per instance. For lighter needs, a plain cron job or a scheduled CI workflow can trigger the producer instead; see Scheduling Scrapers with Cron and GitHub Actions.

Related

• Scaling Python Web Scrapers — how distribution compares with the other scaling routes
• Celery vs RQ for Scraping Task Queues — a lighter alternative and when it is enough
• Monitoring and Alerting for Scrapers — queue depth, success rate and failure alerts
• Deploying Scrapers to the Cloud — running the worker fleet somewhere durable
• Deduplicating URLs with Bloom Filters — constant-memory frontier deduplication