Celery vs RQ for Scraping Task Queues

Once a crawl outgrows one process, the next decision is which Python job queue runs the workers, and this page — part of Distributed Crawling with Celery and Redis — compares Celery and RQ on the terms that matter to a scraper rather than to a generic web backend.

Start with RQ if you want a Redis-only queue small enough to read end to end, your volume is moderate, and your tasks are plain functions. Move to Celery when the crawl needs per-task rate limiting, routing across pools with different resource profiles, a built-in recurring scheduler, or a broker with stronger delivery guarantees than Redis provides. Both distribute work correctly; the difference is how much of the surrounding machinery you have to build, and how much operational surface you take on in exchange.

The Four Features That Actually Decide It

Most feature-comparison tables list a dozen dimensions that make no difference to a crawl. In practice four do, and on all four Celery ships behaviour that RQ asks you to assemble.

Per-task rate limiting is the one that bites first. Scraping is the rare workload where going slower is a requirement, not a regression. Celery's @app.task(rate_limit="10/m") throttles at the worker before the task body runs; in RQ you write your own token bucket in Redis and call it at the top of every task, which works but is now your code to test.

Routing matters as soon as tasks stop being uniform. A crawl usually has cheap HTTP fetches that you want hundreds of in flight and browser renders that consume hundreds of megabytes each. Those belong on separate queues consumed by separately-sized worker pools. RQ supports multiple queues and a worker can listen to several in priority order, so you can build this; Celery adds declarative task_routes so the routing lives in configuration rather than at every call site.

A recurring schedule is Celery Beat, which ships in the box, versus rq-scheduler, a separate package and a separate long-running process. Both work. Beat is one fewer thing to deploy.

Broker choice is the hard constraint. RQ is Redis-only by design. If your organisation already runs RabbitMQ, or you need publisher confirms and per-message acknowledgement semantics that Redis lists do not give you, RQ is out of the running before the comparison starts.

Working Through the Choice

The order matters. Requirements that RQ cannot meet at all — a non-Redis broker, or rate limiting you are not willing to hand-roll — settle the question immediately. Everything else is a preference for a smaller library over a larger one, and that preference is worth taking seriously: RQ's entire source is a few thousand lines, and when a job wedges you can read the code that wedged it. Celery's configuration surface is large enough that most teams eventually hit a behaviour they did not know they had enabled.

Concurrency Models and Throughput

Both push work through Redis and both scale horizontally by adding processes and machines, so the throughput difference is not about the broker. It is about how each executes tasks inside a worker.

RQ forks a subprocess per job. Every task gets a clean interpreter state, so a segfault in a C parser or a leaked file handle dies with the job rather than accumulating. The cost is the fork itself — a few milliseconds plus copy-on-write page faults — which is irrelevant for a task that spends 400 ms waiting on a socket and significant for one that takes 5 ms. RQ also ships SimpleWorker, which runs jobs in the main process without forking, and worker pools that run several such workers together.

Celery gives you a choice of pool. The prefork default behaves much like RQ, with a fixed set of child processes rather than a fork per task. The interesting option for scraping is gevent or eventlet: a single worker process that monkey-patches the socket library and keeps hundreds of I/O-bound fetches in flight cooperatively.

That last point is the real throughput story. A network-bound fetch task spends nearly all its time blocked on a socket, so a gevent pool with --concurrency=200 can keep 200 requests in flight inside one process using a few hundred megabytes, where prefork would need 200 processes. Indicatively, on a modest 2 vCPU box fetching pages with a 500 ms average response time, a prefork pool of 8 sustains roughly 16 pages per second while a gevent pool of 200 in the same process can sustain far more — until the target's rate limit, not your worker, becomes the constraint. Treat those as illustrative arithmetic from the response-time figure rather than a measured benchmark; the real ceiling on a scraper is almost always politeness, and the concurrency you should run is the number the target tolerates.

Two warnings about gevent. It requires that every library in the task be patchable pure Python or gevent-aware; a C extension that blocks the socket directly will stall the whole worker. And it must not be used for CPU-bound work — a heavy parse blocks every other greenlet in the process. The standard arrangement is a gevent pool for fetching and a prefork pool for parsing and rendering, which is exactly the routing case above.

The Same Task in Both

The day-to-day code is close enough that porting is mostly mechanical. Here is the RQ version — enqueue from anywhere, run with rq worker fetch.

# rq_version.py — run the worker with: rq worker fetch
import httpx
from redis import Redis
from rq import Queue, Retry

HEADERS = {
    "User-Agent": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 "
    "(KHTML, like Gecko) Chrome/125.0 Safari/537.36",
    "Accept": "text/html,application/xhtml+xml",
}


def fetch(url: str) -> dict[str, int]:
    with httpx.Client(timeout=15.0, follow_redirects=True, headers=HEADERS) as client:
        response = client.get(url)
    response.raise_for_status()
    return {"status": response.status_code, "length": len(response.text)}


if __name__ == "__main__":
    queue = Queue("fetch", connection=Redis(host="127.0.0.1", port=6379, db=0))
    job = queue.enqueue(
        fetch,
        "https://books.toscrape.com/",
        job_timeout=60,
        result_ttl=300,
        retry=Retry(max=3, interval=[10, 30, 60]),
    )
    print(f"enqueued {job.id}")

Retry(max=3, interval=[...]) gives explicit backoff intervals rather than a formula, and result_ttl=300 stops return values living in Redis forever — the default is 500 seconds, which is fine, but leaving it unset on a million-job crawl is how a Redis instance fills up.

The Celery version, run with celery -A celery_version worker -Q fetch -P gevent -c 100:

# celery_version.py
import httpx
from celery import Celery

app = Celery(
    "scraper",
    broker="redis://127.0.0.1:6379/0",
    backend="redis://127.0.0.1:6379/1",
)
app.conf.update(
    task_acks_late=True,
    worker_prefetch_multiplier=1,
    result_expires=300,
    task_routes={"celery_version.render": {"queue": "render"}},
)

HEADERS = {
    "User-Agent": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 "
    "(KHTML, like Gecko) Chrome/125.0 Safari/537.36",
    "Accept": "text/html,application/xhtml+xml",
}


@app.task(bind=True, max_retries=3, rate_limit="10/s", autoretry_for=(httpx.HTTPError,),
          retry_backoff=2, retry_jitter=True)
def fetch(self, url: str) -> dict[str, int]:
    with httpx.Client(timeout=15.0, follow_redirects=True, headers=HEADERS) as client:
        response = client.get(url)
    response.raise_for_status()
    return {"status": response.status_code, "length": len(response.text)}


if __name__ == "__main__":
    fetch.apply_async(args=["https://books.toscrape.com/"], queue="fetch")

The decorator carries the throttle and the retry policy declaratively, and autoretry_for with retry_backoff and retry_jitter reproduces the exponential-backoff-with-jitter behaviour described in Retrying Failed Requests with Tenacity without a second library.

The two configuration lines above the tasks matter more than they look. task_acks_late=True acknowledges a message only after the task completes, so a worker killed mid-fetch causes redelivery instead of silent loss — essential for a crawl you cannot easily re-derive. worker_prefetch_multiplier=1 stops a worker reserving a batch of messages it will sit on for minutes, which is what makes an uneven fetch queue distribute evenly. The equivalent in RQ is that a job is only removed from its queue when the worker picks it up and is moved to a registry, with job_timeout governing when a stalled job is failed.

Operational Cost

The queue you can debug at 2am is the one that stays. Two practical differences:

Failure visibility. RQ's failed-job registry keeps the exception and traceback attached to the job, and rq info gives you a live count per queue. Celery has richer tooling — Flower, events, inspect commands — but you have to run it, and the default configuration keeps no result at all for tasks that return None.

Memory behaviour. Long-running Celery prefork children accumulate memory from libraries that leak (lxml on malformed documents is a classic), so production deployments usually set worker_max_tasks_per_child to recycle them. RQ's fork-per-job model makes that class of leak structurally impossible, which is a real operational advantage for parsers you do not control.

Whichever you pick, the queue distributes work but does not track what has already been crawled — that needs a deduplication layer such as the one in Deduplicating URLs with Bloom Filters — and it does not hide your egress addresses, so pair either with Rotating Proxies and Managing IP Blocks.

Edge Cases and Caveats

• RQ is Redis-only, permanently. It is a design decision, not a gap. If you need RabbitMQ's acknowledgement and routing semantics, Celery is the only candidate of the two.
• RQ needs os.fork. It does not run natively on Windows workers. Celery runs on Windows but with pool restrictions and is not officially supported there; use Linux or containers for either in production.
• Redis as a broker can lose work. Redis is not a durable message broker by default. With task_acks_late and an appropriate visibility_timeout the exposure is small, but a Redis instance without persistence that restarts loses whatever was queued. Size your tolerance for that before choosing Redis over RabbitMQ.
• Result backends grow silently. Both can store return values in Redis. Return an identifier, not a page — write real data to a database from inside the task, as in Storing and Exporting Scraped Data — and set result_ttl or result_expires.
• Neither renders JavaScript. A queue distributes work; it does not fetch dynamic pages. Run the browser inside the task on a separate, much smaller pool, because each context costs orders of magnitude more memory than an HTTP fetch.
• Celery 5 dropped some Python 2-era idioms. Task decorators, the CLI (celery -A app worker rather than celery worker -A app) and configuration key names all changed at 4.x to 5.x. Old tutorials will not run; check the version before copying configuration.
• Rate limits are per worker, not global. Celery's rate_limit applies to each worker process independently, so ten workers with 10/s allow 100 requests per second in total. For a true global limit you still need a shared token bucket in Redis.

Frequently Asked Questions

Is RQ fast enough for serious scraping? For thousands to low millions of URLs, yes. Its fork-per-job overhead is a few milliseconds against fetch tasks that spend hundreds of milliseconds on the network, so dispatch is rarely the bottleneck, and you scale by adding workers and machines. The overhead only starts to matter for very short tasks at very high rates.

When is Celery clearly the better choice? When you need per-task rate limiting, declarative routing across pools with different resource profiles, workflow primitives such as chains and chords, a broker other than Redis, or a scheduler that ships with the framework. Those are the features a large, long-running crawl tends to grow into, and building each of them on top of RQ costs more than adopting Celery would have.

Can I migrate from RQ to Celery later? Yes, and it is a common path. Task bodies are ordinary Python functions in both, so the fetch and parse logic ports unchanged; what you rewrite is the enqueue calls, the worker startup command and the deployment configuration. Starting small and migrating when a hard requirement appears is a defensible strategy.

Do I still need rotating proxies with either queue? Yes. A queue distributes tasks across workers, but those workers usually share a handful of egress IP addresses, so from the target's perspective the traffic still comes from one place — only faster. Concurrency without IP rotation and rate limiting is the quickest route to a block.

Related

• Distributed Crawling with Celery and Redis — the parent topic covering brokers, workers and crawl frontiers.
• Limiting Concurrency with Semaphores — the in-process equivalent when a queue would be overkill.
• Structured Logging for Python Scrapers — making worker output searchable once tasks run on many machines.
• Deploying Scrapers to the Cloud — where the workers and the broker actually run.