httpx vs aiohttp Async Performance

Picking an async HTTP client sets the ceiling on how fast a crawler can go, and this page — part of Asynchronous Scraping with Asyncio and HTTPX — compares httpx and aiohttp on the axes that actually decide a scraping project.

For raw requests per second over HTTP/1.1, aiohttp is usually a little faster, because it is a single-purpose async client with a shorter code path and a C-accelerated parser. httpx gives up a few percent of that speed for a shared sync and async API, native HTTP/2, and a transport layer you can swap out in tests. For nearly every real crawl the difference disappears into network latency and politeness delays, so the deciding factors are protocol support and ergonomics, not the microbenchmark. Pick aiohttp when you are hammering one fast endpoint you control; pick httpx when you want HTTP/2, a requests-shaped API, or one client class that works in both a script and a service.

How the Two Clients Are Built

The performance difference is not a mystery — it falls out of the architecture. aiohttp was designed as an asyncio-only library. Its ClientSession talks to a TCPConnector, which talks to an HTTP/1.1 parser with a C extension behind it, which talks to a socket. Four layers, all of which assume a running event loop.

httpx has a different goal: httpx.Client and httpx.AsyncClient expose the same methods, the same Response object, and the same timeout and limits configuration. That symmetry is only possible because both delegate to a shared transport package, httpcore, which then dispatches to h11 for HTTP/1.1 or h2 for HTTP/2. The extra indirection is what you pay for the shared surface.

That transport boundary is also httpx's best feature for testing. Because every request goes through a transport object, you can pass httpx.MockTransport into a client and run a spider's parsing logic against canned responses with no network at all. aiohttp has no equivalent seam, so its tests usually spin up a real local server through aiohttp.test_utils.

Where the Performance Gap Comes From

Three things account for the measurable difference:

• Response construction. httpx builds a richer Response object and eagerly decodes and validates more of it. That catches malformed responses earlier, and it costs microseconds per request.
• The transport hop. Every httpx request crosses the httpcore boundary, including its own connection-pool bookkeeping, before reaching the protocol layer.
• Parser implementation. aiohttp ships a C-accelerated HTTP parser (falling back to a pure-Python one if the extension is unavailable); h11 is pure Python by design.

To put a size on it: on a laptop fetching a 2 KB JSON body from a local server on loopback, with 50 concurrent tasks and no per-request delay, aiohttp typically completes a fixed batch somewhere in the region of 10–25% faster than httpx on HTTP/1.1. That is an indicative figure from a synthetic setup with no network in the path, not an authoritative benchmark — the point is the order of magnitude of the difference, which is "small". Run the same comparison against a real site 80 ms away and the two are indistinguishable, because 80 ms of round-trip time dwarfs 200 microseconds of client overhead.

The practical rule: if your bottleneck is the target server, the client choice does not change your throughput. If your bottleneck is your own CPU — because you are pulling from a private API on the same network at tens of thousands of requests per minute — then aiohttp buys you real headroom, and you should also check whether your parsing, not your fetching, is the actual constraint. Measuring that split is one of the jobs of Monitoring and Alerting for Scrapers.

Sessions, Pools and the One Mistake That Matters

Both libraries reuse TCP connections through a keep-alive pool, and in both the single most expensive mistake is creating a client per request. A fresh client means a fresh pool, so every call redoes DNS resolution, the TCP handshake and the TLS handshake — typically 100–300 ms of pure overhead against a remote host, versus roughly zero for a reused connection. The client is meant to live for the whole run.

The second thing to get right is the relationship between the pool limit and the concurrency limit. They are different knobs:

• The pool caps how many sockets can be open to a host at once. In aiohttp that is TCPConnector(limit=..., limit_per_host=...); in httpx it is httpx.Limits(max_connections=..., max_keepalive_connections=...).
• The semaphore caps how many coroutines are in flight. Without one, asyncio.gather over 50,000 URLs schedules 50,000 coroutines immediately; they will queue on the pool rather than open 50,000 sockets, but every one of them still holds its own task, request object and future in memory.

Set both. The full pattern, including how to size the semaphore against a target's rate limits, is covered in Limiting Concurrency with Semaphores. An uncapped async crawler is also the fastest possible way to earn an IP ban, which is the failure mode Rotating Proxies and Managing IP Blocks exists to manage.

A Runnable Comparison

The two clients are close enough in shape that the same crawl reads almost identically in each. Here is the aiohttp version, with a bounded connector, an explicit total timeout, a semaphore and a realistic User-Agent.

import asyncio
import time

import aiohttp

HEADERS = {
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 "
    "(KHTML, like Gecko) Chrome/125.0 Safari/537.36",
    "Accept": "application/json",
}


async def fetch(
    session: aiohttp.ClientSession, sem: asyncio.Semaphore, url: str
) -> tuple[str, int, int]:
    async with sem:
        async with session.get(url, headers=HEADERS) as resp:
            body = await resp.read()
            return url, resp.status, len(body)


async def crawl(urls: list[str], concurrency: int = 10) -> list[tuple[str, int, int]]:
    sem = asyncio.Semaphore(concurrency)
    connector = aiohttp.TCPConnector(limit=concurrency, ttl_dns_cache=300)
    timeout = aiohttp.ClientTimeout(total=30, connect=10)
    async with aiohttp.ClientSession(connector=connector, timeout=timeout) as session:
        return await asyncio.gather(*(fetch(session, sem, u) for u in urls))


if __name__ == "__main__":
    sample = ["https://httpbin.org/get"] * 20
    started = time.perf_counter()
    results = asyncio.run(crawl(sample))
    elapsed = time.perf_counter() - started
    print(f"aiohttp: {len(results)} pages in {elapsed:.2f}s, first status {results[0][1]}")

The httpx equivalent swaps three names and adds one argument:

import asyncio
import time

import httpx

HEADERS = {
    "User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 "
    "(KHTML, like Gecko) Version/17.4 Safari/605.1.15",
    "Accept": "application/json",
}


async def fetch(
    client: httpx.AsyncClient, sem: asyncio.Semaphore, url: str
) -> tuple[str, int, int]:
    async with sem:
        resp = await client.get(url, headers=HEADERS)
        return url, resp.status_code, len(resp.content)


async def crawl(urls: list[str], concurrency: int = 10) -> list[tuple[str, int, int]]:
    sem = asyncio.Semaphore(concurrency)
    limits = httpx.Limits(max_connections=concurrency, max_keepalive_connections=concurrency)
    timeout = httpx.Timeout(30.0, connect=10.0)
    async with httpx.AsyncClient(limits=limits, timeout=timeout, http2=True) as client:
        results = await asyncio.gather(*(fetch(client, sem, u) for u in urls))
    return results


if __name__ == "__main__":
    sample = ["https://httpbin.org/get"] * 20
    started = time.perf_counter()
    out = asyncio.run(crawl(sample))
    elapsed = time.perf_counter() - started
    print(f"httpx: {len(out)} pages in {elapsed:.2f}s, first status {out[0][1]}")

Run both against the same list on the same connection before you commit to either. A five-minute measurement against your actual targets beats any published benchmark, because the answer depends on how far away the server is and how much work your parser does per page.

HTTP/2 and the Protocol Difference

This is the clearest functional split. httpx speaks HTTP/2 when you install the extra (pip install "httpx[http2]") and pass http2=True. Over HTTP/2 a single connection multiplexes many concurrent streams, so 20 concurrent requests to one host need one socket and one TLS handshake instead of 20. Against a CDN-fronted site that can cut connection setup out of the picture entirely.

aiohttp's stable releases are HTTP/1.1 only. If your targets are HTTP/2 and you want multiplexing, httpx is the straightforward answer.

There is a second, less obvious consequence. Negotiating HTTP/2 changes your TLS ClientHello and your header ordering, so a client that speaks HTTP/2 presents a different network fingerprint from one that does not — and on a site that expects browser traffic, HTTP/1.1 from a "Chrome" User-Agent is itself a signal. That interaction is the subject of TLS and JA3 Fingerprint Evasion. Verify what you actually negotiated with response.http_version, which returns the string "HTTP/2" or "HTTP/1.1"; a silent fallback to HTTP/1.1 when the h2 package is missing is easy to miss.

On ergonomics, httpx wins for teams coming from requests — the same .json(), the same params=, the same status codes, plus proper timeout objects and follow_redirects as an explicit flag rather than a default. aiohttp wins if you also need a server: it ships a full web framework and WebSocket support, so the control-plane API around your crawler can live in the same dependency.

Edge Cases and Caveats

• follow_redirects defaults differ. httpx does not follow redirects unless you ask; aiohttp does by default. Porting code between them without noticing produces a pile of unexplained 301 responses.
• Timeouts mean different things. aiohttp.ClientTimeout(total=30) covers the whole operation including reading the body. httpx.Timeout(30.0) sets connect, read, write and pool timeouts to 30 s each, so a slow-drip response can outlive it. Set connect explicitly in both.
• resp.text() triggers charset detection. In aiohttp, calling await resp.text() on a page with a mislabeled or absent charset can raise UnicodeDecodeError or silently mojibake the content. Read await resp.read() and decode deliberately — see Fixing Common Unicode Errors in Python Scraping.
• The response body must be consumed inside the aiohttp context. Leaving async with session.get(...) before reading the body releases the connection and invalidates the response. httpx reads the body eagerly by default, so it has no such rule unless you use client.stream().
• DNS caching differs. aiohttp caches resolutions itself via ttl_dns_cache; httpx defers to the OS resolver, which on some Linux containers means a real lookup per connection. Under heavy fan-out to many hosts, DNS becomes the bottleneck before either client does.
• HTTP/2 needs the extra. http2=True without pip install "httpx[http2]" raises ImportError on client construction in current versions; older ones fell back silently. Assert on response.http_version in a smoke test either way.
• Proxy configuration is not portable. httpx takes proxy= (singular, since 0.26) on the client; aiohttp takes proxy= per request, and neither reads the same environment variables in the same way. Test your proxy path explicitly after switching.

Frequently Asked Questions

Is aiohttp always faster than httpx? On HTTP/1.1 against a fast, nearby endpoint it is usually a little faster, in the region of 10–25% on a synthetic loopback test. Once real network latency and polite per-domain delays are in the path, the difference is far smaller than the run-to-run variance, so it should rarely decide the choice.

Should I pick httpx just for HTTP/2? If your targets serve HTTP/2 and you want connection multiplexing or a more browser-like network profile, yes — aiohttp's stable line is HTTP/1.1 only, so httpx is the only one of the two that can do it. Install the h2 extra and confirm with response.http_version rather than assuming.

Can I share one client across many coroutines? Yes, and you should. Both httpx.AsyncClient and aiohttp.ClientSession are built to be created once and reused for every request in a run so the connection pool, keep-alive and DNS cache can do their jobs. Creating one per request is the most common and most expensive async scraping mistake.

Which is the easier migration from requests?httpx, comfortably. Its API mirrors requests closely enough that most call sites port unchanged, and it offers both a blocking Client and an AsyncClient with the same surface, so you can convert a synchronous scraper first and switch to async once the parsing logic is proven.

Related

• Asynchronous Scraping with Asyncio and HTTPX — the parent topic, from event loop basics to a working async crawler.
• Limiting Concurrency with Semaphores — how to size the in-flight limit that sits in front of either client.
• Retrying Failed Requests with Tenacity — the retry policy both clients need around them.
• HTTP Caching with requests-cache — the cheapest speedup of all: not sending the request.