How to Rotate User Agents in Python

Changing the User-Agent header is the first thing most people try, and it is the shallowest of the signals discussed in Rotating Proxies and Managing IP Blocks.

Rotating the User-Agent addresses one shallow signal; anti-bot systems also weigh companion headers, TLS fingerprint, and IP reputation.

Build a small pool of genuine, current browser strings, store each one with the companion headers that browser actually sends, and select a whole profile — never a bare string — per session rather than per request. In requests or httpx that means passing the full header mapping and keeping it stable for the lifetime of the client. Done properly this removes one obvious tell. Done as commonly described, with a list of a thousand scraped strings picked at random on every call, it creates a new tell that is easier to spot than the default python-requests/2.32.3 you started with.

The obligations that come with the technique do not change: a rotating header does not extend your permission to fetch anything. Honour robots.txt, keep within any documented rate limit, and stop when a site asks you to.

What the Detector Reads Alongside the String

A User-Agent is a claim. Everything else in the request is evidence, and a detector's job is to check whether the evidence supports the claim.

A detector does not read the User-Agent in isolation — it checks whether the client hints and the TLS handshake tell the same story. Disagreement is the signal, not the header itself.

Four checks account for most real-world flagging, and each one is cheap for a server to run:

Client hints must agree with the string. Since Chrome 89, Chromium browsers send Sec-CH-UA, Sec-CH-UA-Mobile and Sec-CH-UA-Platform on every request. A User-Agent claiming Chrome 125 that arrives without any Sec-CH-UA header is a contradiction that requires no machine learning to catch — the browser it names always sends them. Equally, a Firefox User-Agent that does send client hints is wrong in the other direction, because Firefox does not implement them.

Sec-Fetch-* metadata must match the navigation. Browsers send Sec-Fetch-Site, Sec-Fetch-Mode, Sec-Fetch-Dest and Sec-Fetch-User describing the context of the request. A top-level document fetch has Sec-Fetch-Dest: document and Sec-Fetch-Mode: navigate; an XHR does not. Sending document metadata for every request in a crawl, including ones that fetch JSON, is inconsistent.

Header order and casing are stable per browser. HTTP/1.1 header order is not semantically meaningful, but it is highly characteristic: Chrome, Firefox and Safari each emit a fixed sequence. Python's requests builds its header block from a case-insensitive dict seeded with its own defaults, and the resulting order matches no browser. httpx behaves similarly. Over HTTP/2 the equivalent signal is pseudo-header order and the SETTINGS frame, which is more distinctive still.

The TLS handshake sits underneath all of it. No header changes which cipher suites and extensions your client offers. A Chrome User-Agent over an OpenSSL handshake is the single most common mismatch in scraping traffic, and fixing it means changing the HTTP client, not the headers — the subject of TLS and JA3 Fingerprint Evasion and, concretely, Using curl_cffi to Impersonate Browsers.

Choosing the Rotation Granularity

Pool size is the wrong dial. Granularity is the right one.

Rotation granularity, not pool size, is what looks human. A browser identity should change when the session changes, never between two requests that share a cookie jar.

A real visit is one browser, one identity, many requests. If your requests share a cookie jar but not a User-Agent, you have described a user who changed browser mid-session while keeping their session cookie — which is impossible, and trivially detectable from the server's access log alone. Rotate at the session boundary, and rotate the whole identity together: the User-Agent, the cookie jar, and the outbound IP. That coupling is why User-Agent rotation is discussed here rather than in isolation, and why it interacts with the pool choice in Residential vs Datacenter Proxies.

Keep the pool small. Five to ten profiles covering the browser and platform combinations that plausibly visit your target is more convincing than a hundred, because a hundred inevitably contains implausible entries — a Chrome 91 on Windows 7, an Android build string that never shipped — and one implausible entry is worse than none.

Building Coherent Profiles

The helper below stores each User-Agent with the headers that browser genuinely sends, so nothing can drift out of sync. A profile is selected once and bound to a client, following the session discipline in Managing Cookies and Sessions.

import random
import requests

UA_PROFILES: list[dict[str, str]] = [
    {
        "User-Agent": (
            "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 "
            "(KHTML, like Gecko) Chrome/125.0.0.0 Safari/537.36"
        ),
        "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8",
        "Accept-Language": "en-US,en;q=0.9",
        "Accept-Encoding": "gzip, deflate, br",
        "Sec-CH-UA": '"Chromium";v="125", "Not.A/Brand";v="24", "Google Chrome";v="125"',
        "Sec-CH-UA-Mobile": "?0",
        "Sec-CH-UA-Platform": '"Windows"',
        "Sec-Fetch-Dest": "document",
        "Sec-Fetch-Mode": "navigate",
        "Sec-Fetch-Site": "none",
        "Upgrade-Insecure-Requests": "1",
    },
    {
        "User-Agent": (
            "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 "
            "(KHTML, like Gecko) Version/17.4 Safari/605.1.15"
        ),
        "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
        "Accept-Language": "en-US,en;q=0.9",
        "Accept-Encoding": "gzip, deflate, br",
        "Upgrade-Insecure-Requests": "1",
    },
    {
        "User-Agent": "Mozilla/5.0 (X11; Linux x86_64; rv:127.0) Gecko/20100101 Firefox/127.0",
        "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8",
        "Accept-Language": "en-US,en;q=0.5",
        "Accept-Encoding": "gzip, deflate, br",
        "Upgrade-Insecure-Requests": "1",
    },
]


def make_session() -> requests.Session:
    """Bind one coherent browser identity to one session for its whole lifetime."""
    session = requests.Session()
    session.headers.clear()
    session.headers.update(random.choice(UA_PROFILES))
    return session


def fetch(session: requests.Session, url: str) -> requests.Response:
    response = session.get(url, timeout=15)
    response.raise_for_status()
    return response


if __name__ == "__main__":
    with make_session() as sess:
        seen = fetch(sess, "https://httpbin.org/headers").json()["headers"]
        print(seen["User-Agent"])
        print(seen.get("Sec-Ch-Ua", "no client hints"))

session.headers.clear() matters more than it looks. Without it, the library's own defaults — including its User-Agent and a permissive Accept: */* — remain underneath your updates, and Accept: */* on a document request is itself unusual.

The httpx version is the same idea with HTTP/2 available, which changes the header transport but not the coherence requirement:

import json
import random
from pathlib import Path
from typing import Any

import httpx

DEFAULT_PROFILES: list[dict[str, str]] = [
    {
        "User-Agent": (
            "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 "
            "(KHTML, like Gecko) Chrome/125.0.0.0 Safari/537.36"
        ),
        "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8",
        "Accept-Language": "en-US,en;q=0.9",
        "Sec-CH-UA": '"Chromium";v="125", "Not.A/Brand";v="24", "Google Chrome";v="125"',
        "Sec-CH-UA-Mobile": "?0",
        "Sec-CH-UA-Platform": '"Windows"',
    },
    {
        "User-Agent": "Mozilla/5.0 (X11; Linux x86_64; rv:127.0) Gecko/20100101 Firefox/127.0",
        "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8",
        "Accept-Language": "en-US,en;q=0.5",
    },
]


def load_profiles(path: str = "ua_profiles.json") -> list[dict[str, str]]:
    """Read profiles from disk if present, otherwise fall back to the built-ins."""
    source = Path(path)
    if source.is_file():
        return json.loads(source.read_text(encoding="utf-8"))
    return DEFAULT_PROFILES


def make_client(profiles: list[dict[str, str]]) -> httpx.Client:
    """One client, one identity, HTTP/2 enabled to match a modern browser."""
    return httpx.Client(
        headers=random.choice(profiles),
        http2=True,
        timeout=15.0,
        follow_redirects=True,
    )


def fetch_json(client: httpx.Client, url: str) -> dict[str, Any]:
    response = client.get(url)
    response.raise_for_status()
    return response.json()


if __name__ == "__main__":
    with make_client(load_profiles()) as client:
        print(fetch_json(client, "https://httpbin.org/headers")["headers"]["User-Agent"])

Loading the pool from a file when one exists is deliberate: the strings are the part that goes stale, and a regenerated data file is easier to review in a diff than a literal buried in a module. http2=True needs the h2 extra (pip install "httpx[http2]"); without it the client silently negotiates HTTP/1.1, which is a different fingerprint from the browser you are claiming to be.

Version Drift and the Errors It Produces

Chrome and Firefox both ship a new stable major roughly every four weeks. A pool written today describes browsers that will be two versions old in two months and eight versions old in eight. Nobody notices, because nothing throws.

What you see instead is a slow change in response mix. Requests that used to return 200 start returning 403 Forbidden from a WAF, or 429 Too Many Requests at a rate you did not previously hit, or — most confusingly — 200 with an interstitial HTML body instead of the content, which raise_for_status() will happily pass through. If your parser starts raising AttributeError: 'NoneType' object has no attribute 'text' on a page that worked last month, check the raw body before you check the selector.

Two practices keep this manageable. Regenerate the profile file whenever you update a browser or a browser-driving dependency, taking the strings from the actual browser rather than from a list site. And log the status-code distribution per run so a drift from 99% 200 to 92% 200 is visible as a trend, rather than being discovered when the numbers go to zero.

There is a second kind of drift worth planning for: the header set itself changes shape over time, not only its values. Client hints did not exist before Chrome 89 and are still being extended; Sec-Fetch-* metadata arrived later than the User-Agent conventions most examples were written against; and the reduced User-Agent scheme freezes the minor version segments at zeroes, so a string with a populated build number now looks older than one without. A profile pool assembled from examples of different vintages will therefore contain entries that are individually plausible but collectively inconsistent — some sending hints, some not, for browsers that would all send them. Regenerate the whole pool at once rather than patching entries individually.

Edge Cases and Caveats

The library default is an instant filter match. python-requests/2.32.3, python-httpx/0.27.0 and Scrapy/2.11 are all blocked outright by common WAF rule sets. Removing them is the entire benefit most scrapers get from this technique.
Mobile strings change the response. An iPhone User-Agent may return a different DOM, a different pagination scheme, or a redirect to an m. host. Only claim mobile if your parser is built for the mobile markup.
Accept-Encoding: br requires Brotli support. If you advertise it without brotli or brotlicffi installed, the response body will be unreadable bytes and you will get a UnicodeDecodeError or silent garbage rather than an obvious error.
Header order is largely out of your hands. requests and httpx both reorder to some degree, and neither reproduces a browser's sequence exactly. If a target is checking order, no amount of pool tuning fixes it; a browser-fingerprinting HTTP client does.
Rotating mid-session breaks more than stealth. Some sites bind a session cookie to the User-Agent that created it and invalidate the session on mismatch, producing a sudden logout rather than a block. Persisting a stable identity across runs, as in Persisting a Session Between Runs, avoids re-triggering that.
A crawler string is sometimes the right answer. If you are collecting on behalf of an identifiable organisation and the target's terms allow it, a descriptive User-Agent naming your bot and a contact URL is more honest, more robust, and frequently gets you allowlisted rather than blocked. Impersonation is for cases where a generic browser identity is genuinely appropriate, not a default.
Async does not change the rule. Each AsyncClient in a pool holds one identity for its lifetime, the same as a synchronous client — see Asynchronous Scraping with Asyncio and HTTPX for the surrounding structure.

Frequently Asked Questions

How do I rotate User-Agent strings in Python? Store complete header profiles rather than bare strings, choose one profile with random.choice, and bind it to a requests.Session or httpx.Client for that client's whole lifetime. Clear the library's default headers first so its own User-Agent and Accept values do not survive underneath yours.

Should the User-Agent change on every request? No. Requests that share a cookie jar should share a User-Agent, because a real visitor does not change browser mid-visit. Rotate at the session boundary and change the User-Agent, the cookies, and the outbound IP together so the whole identity turns over at once.

Why am I still blocked after rotating User-Agents? Because the header is the shallowest signal in the request. IP reputation, the TLS handshake, header order, and request pacing are all unchanged by it. If a target blocks a coherent, current browser profile, the cause is almost always the connection rather than the header.

Where should realistic User-Agent strings come from? From browsers you actually run, or from a maintained package that tracks current releases. Avoid scraped list sites: they accumulate strings that were never real, and one implausible entry in a pool is a clearer signal than having no pool at all.

Rotating Proxies and Managing IP Blocks — the parent topic, and the signal that matters more than this one.
Best Free and Paid Proxy Providers for Scraping — choosing the IP pool the identity rides on.
Understanding HTTP Requests and Responses — what each of these headers means.
How to Scrape a Static Website Without Getting Blocked — the low-profile approach these headers belong to.