Reverse-Engineering Private APIs in Python

Most modern sites do not embed their data in the HTML you first download: the page arrives nearly empty and JavaScript fetches the real content from a private, undocumented JSON API. This guide is part of Data Extraction Patterns and APIs and covers calling those endpoints directly — discovering them in the browser Network tab, replaying the requests in Python with httpx, reproducing the headers and tokens that make them work, and detecting the day the contract changes underneath you.

When to Use This Approach

Reach for private-API scraping when the signals point to a data-driven frontend:

• The page loads content after the initial HTML. A spinner followed by data is the visible symptom of a fetch call.
• View-source shows no data. If prices or listings are missing from the raw HTML but visible in the browser, they arrive over the network afterwards.
• The Network tab shows JSON. Filtering to Fetch/XHR reveals responses with Content-Type: application/json.
• You want throughput. One JSON request replaces a full browser render plus HTML parsing. In practice that is a 20–100× reduction in bytes transferred and a comparable reduction in CPU, because you skip layout, paint and JavaScript execution entirely.
• You need fields the page does not display. Internal APIs routinely return stock levels, internal IDs, cost prices and warehouse codes that the UI never renders.

Do not take this route when the data is already baked into the initial HTML — parse it directly using the techniques in Understanding HTTP Requests and Responses. When the site uses a single GraphQL endpoint rather than plain REST, the query and pagination mechanics differ enough to warrant the separate walkthrough in Scraping GraphQL Endpoints. And when the interface you want belongs to a phone app rather than a website, the interception setup is different again — see Scraping Mobile App APIs.

Prerequisites

You need Python 3.10 or newer and a modern HTTP client. httpx is used throughout because it speaks HTTP/2, which matters more than it sounds: many backends behind a modern CDN answer HTTP/1.1 clients differently, and a client that negotiates HTTP/2 blends in with browser traffic.

python -m pip install "httpx[http2]>=0.27"

The [http2] extra pulls in h2; without it, httpx.Client(http2=True) raises ImportError: Using http2=True, but the 'h2' package is not installed. A Chromium-based browser (Chrome, Edge or Brave) gives you the best Network panel and a "Copy as cURL" option that captures every header verbatim.

Step-by-Step: From Network Tab to Repeatable Client

1. Open the Network Tab and Reproduce the Action

Open DevTools, switch to the Network panel, and filter to Fetch/XHR. Clear the log, then perform the action that loads the data you want — scroll the list, click a tab, submit a search. Each request that appears is a candidate; click one and read the Response sub-tab. If it contains the JSON you are after, you have found your endpoint.

Record four things: the URL including its query string, the method, the request headers, and — for a POST — the request body. The full mechanics of filtering, searching response bodies, and intercepting traffic DevTools cannot see are covered in Finding Hidden API Endpoints in Network Traffic.

2. Replay the Request With the Smallest Possible Client

Start with the URL and a realistic User-Agent. Many public-facing endpoints need nothing more, and starting minimal tells you exactly which header is load-bearing when you add them back.

import httpx

UA = ("Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 "
      "(KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36")


def fetch_json(url: str) -> dict:
    headers = {"User-Agent": UA, "Accept": "application/json"}
    with httpx.Client(http2=True, timeout=15) as client:
        response = client.get(url, headers=headers)
        response.raise_for_status()
        return response.json()


if __name__ == "__main__":
    data = fetch_json("https://httpbin.org/json")
    print(data["slideshow"]["title"])

If this returns your JSON, you are done. A 401, 403, or an HTML error page means the endpoint expects more context.

3. Add the Headers That Actually Matter

Backends distinguish browser traffic from scripts with a small set of headers. Copy them from the captured request and add them one at a time, keeping only the ones that change the outcome.

• Referer — many endpoints reject requests that did not originate from their own pages.
• Origin — checked on cross-origin POST requests, and often required alongside Referer.
• X-Requested-With: XMLHttpRequest — a legacy marker several backends still gate on.
• Accept-Language — sometimes selects the response locale, and occasionally gates access entirely.
• A site-specific header — X-Api-Key, X-Client-Version, X-Store-Id. These are the ones that most often turn a 403 into a 200.

import httpx


def fetch_with_context(url: str) -> dict:
    headers = {
        "User-Agent": ("Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 "
                       "(KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36"),
        "Accept": "application/json, text/plain, */*",
        "Accept-Language": "en-US,en;q=0.9",
        "Referer": "https://www.example.com/products",
        "Origin": "https://www.example.com",
        "X-Requested-With": "XMLHttpRequest",
    }
    with httpx.Client(http2=True, timeout=15) as client:
        response = client.get(url, headers=headers)
        response.raise_for_status()
        return response.json()

Header order occasionally matters too. Some anti-bot layers fingerprint the ordering of a client's headers against known browser profiles; httpx preserves the order of the dict you pass, so listing them in the same order the browser sent them costs nothing and removes one variable. If the request still fails while an identical cURL command succeeds, the discriminator is below the HTTP layer — the TLS handshake — which is the subject of TLS and JA3 Fingerprint Evasion.

4. Reproduce the Authentication Scheme

Endpoints behind a login use one of four shapes, in ascending order of effort.

Cookie-based sessions are the friendliest: log in once and reuse the client so the cookie jar persists, exactly as described in Managing Cookies and Sessions. Bearer tokens require you to find the call that mints them and re-run it when the current one expires.

import base64
import json
import time
import httpx

UA = ("Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 "
      "(KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36")


def jwt_expiry(token: str) -> float | None:
    """Read the exp claim without verifying the signature."""
    try:
        payload = token.split(".")[1]
        payload += "=" * (-len(payload) % 4)          # restore base64url padding
        return float(json.loads(base64.urlsafe_b64decode(payload))["exp"])
    except (IndexError, ValueError, KeyError):
        return None


class ApiSession:
    def __init__(self, login_url: str, username: str, password: str) -> None:
        self._login_url = login_url
        self._username = username
        self._password = password
        self._expires_at = 0.0
        self.client = httpx.Client(
            http2=True, timeout=15,
            headers={"User-Agent": UA, "Accept": "application/json"},
        )

    def _login(self) -> None:
        resp = self.client.post(
            self._login_url,
            json={"username": self._username, "password": self._password},
        )
        resp.raise_for_status()
        token = resp.json()["access_token"]
        self.client.headers["Authorization"] = f"Bearer {token}"
        self._expires_at = jwt_expiry(token) or (time.time() + 300)

    def get(self, url: str, **kwargs: object) -> dict:
        if time.time() > self._expires_at - 30:        # refresh 30s before expiry
            self._login()
        resp = self.client.get(url, **kwargs)
        if resp.status_code == 401:                    # server disagreed; force one retry
            self._login()
            resp = self.client.get(url, **kwargs)
        resp.raise_for_status()
        return resp.json()

Reading the exp claim turns token refresh from reactive into proactive. The reactive path is still there as a fallback because clock skew and server-side revocation both exist, but pre-emptive refresh means a long crawl does not lose a request every fifteen minutes. Note that this decodes the payload without verifying the signature — that is correct here, because you are a client reading a token you were given, not a server authenticating one. OAuth authorization-code flows, where the token comes from a redirect rather than a JSON login, are covered in Scraping Behind Bearer Tokens and OAuth.

The fourth shape — a signed parameter such as ?sig=8f2b…&ts=1735689600 — has no clean solution. Either read the site's bundled JavaScript and port the signing function to Python, or drive a real browser and let it compute the value for you. Before investing hours in the first option, check whether the signature actually validates: a surprising number of sites compute one and never check it server-side.

5. Paginate to the End

Private APIs almost always paginate, in one of three styles: numbered pages (?page=2), offset and limit (?offset=40&limit=20), or an opaque continuation token echoed back from the previous response. Numbered and offset pagination share a failure mode — if the underlying list changes between requests, records shift across page boundaries and you silently skip or duplicate them.

import httpx

UA = ("Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 "
      "(KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36")


def scrape_all_pages(base_url: str, page_size: int = 50, max_pages: int = 500) -> list[dict]:
    headers = {"User-Agent": UA, "Accept": "application/json"}
    results: list[dict] = []
    seen_ids: set[object] = set()
    with httpx.Client(http2=True, timeout=15, headers=headers) as client:
        for page in range(1, max_pages + 1):
            response = client.get(base_url, params={"page": page, "limit": page_size})
            response.raise_for_status()
            payload = response.json()
            items = payload.get("items", [])
            if not items:
                break
            fresh = [i for i in items if i.get("id") not in seen_ids]
            seen_ids.update(i.get("id") for i in fresh)
            results.extend(fresh)
            if len(items) < page_size or not payload.get("has_more", True):
                break
    return results

Three guards earn their place. max_pages stops a runaway loop when a server answers has_more: true forever. The seen_ids set makes shifting boundaries idempotent rather than corrupting. And len(items) < page_size catches the common case of a final short page on an API that omits has_more entirely. Once you have the raw JSON, the reshaping patterns live in Parsing JSON and XML Responses, and the field-level coercion in Cleaning and Validating Scraped Data.

6. Detect the Day the Contract Changes

An undocumented endpoint carries no compatibility promise, but the failure modes are not evenly distributed. Additive changes vastly outnumber breaking ones, so a parser that reads named fields and ignores everything else survives most of them.

The dangerous change is the silent one: a renamed pagination argument that makes every page return the first page, or a filter parameter the server starts ignoring. Neither raises an exception. The cheap defence is an assertion on the shape of what you received, run every crawl:

REQUIRED_FIELDS = {"id", "name", "price"}


def assert_contract(items: list[dict], expected_min: int = 1) -> None:
    if len(items) < expected_min:
        raise RuntimeError(f"contract check: expected >= {expected_min} items, got {len(items)}")
    missing = REQUIRED_FIELDS - set(items[0])
    if missing:
        raise RuntimeError(f"contract check: response is missing {sorted(missing)}")

Wire that into your run and alert on it. A scraper that returns zero rows is easy to notice; a scraper that returns the same fifty rows every night for a month is not.

7. Reproduce POST Bodies and Rate Limits Correctly

Not every private endpoint is a GET. Search, filter and batch endpoints usually POST, and the body format is the detail most often reproduced incorrectly. Three encodings appear in practice, and each maps to a different httpx argument:

• application/json — pass json=payload. httpx serialises it and sets the header for you. Passing data=json.dumps(payload) instead sends the correct bytes with the wrong content type and produces a 415 Unsupported Media Type.
• application/x-www-form-urlencoded — pass data={"q": "chairs", "page": 1}. This is what a classic form submits and what several older backends still expect.
• multipart/form-data — pass files={...}. Rare for data endpoints, common for anything that also accepts an upload.

Getting the encoding right is not enough on its own if the server enforces limits. A private API sized for interactive use will start rejecting a crawler long before a public API would, and the well-behaved response is to slow down rather than retry harder:

import time
import httpx

UA = ("Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 "
      "(KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36")


def post_with_backoff(client: httpx.Client, url: str, payload: dict,
                      attempts: int = 5) -> dict:
    delay = 1.0
    for attempt in range(attempts):
        response = client.post(url, json=payload)
        if response.status_code == 429:
            wait = float(response.headers.get("Retry-After", delay))
            time.sleep(min(wait, 60.0))
            delay = min(delay * 2, 60.0)
            continue
        if response.status_code >= 500:
            time.sleep(delay)
            delay = min(delay * 2, 60.0)
            continue
        response.raise_for_status()
        return response.json()
    raise RuntimeError(f"gave up on {url} after {attempts} attempts")


def search(query: str) -> dict:
    headers = {"User-Agent": UA, "Accept": "application/json",
               "Origin": "https://www.example.com",
               "Referer": "https://www.example.com/search"}
    with httpx.Client(http2=True, timeout=20, headers=headers) as client:
        return post_with_backoff(client, "https://www.example.com/api/search",
                                 {"q": query, "page": 1, "size": 50})

Two rules are encoded here. Retry-After is honoured when present, because the server has told you exactly how long it wants — sleeping less earns a longer penalty and sleeping more wastes time. And 4xx responses other than 429 are not retried: a 400 or 403 means the request itself is wrong, and repeating it identically will fail identically while making your traffic look like a brute-force attempt.

Performance and Scaling Considerations

Hitting a JSON API is dramatically cheaper than driving a browser, and that efficiency makes it tempting to hammer the endpoint. Pace yourself deliberately.

• Reuse one Client. Connection pooling saves the TCP and TLS handshake on every request after the first — typically 100–300 ms per request on a cross-continental link, which dominates the response time of a fast endpoint.
• Concurrency belongs behind a semaphore. Private APIs are frequently sized for real user traffic, not for a crawler. The bounded pattern is in Asynchronous Scraping with asyncio and HTTPX; eight to sixteen concurrent requests is a sane starting ceiling.
• Honour Retry-After. A 429 response usually carries it. Sleeping the advertised interval is both correct and faster than backing off blindly, because it avoids a second penalty.
• Cache during development. Re-fetching the same page while iterating on parsing logic wastes your time and the site's capacity; a transparent HTTP cache as described in HTTP Caching with requests-cache makes the second run instant.
• Measure with the endpoint's own numbers. Log the response size and elapsed time per request. A response that suddenly halves in size, or an endpoint whose latency doubles under your load, tells you more about the right concurrency ceiling than any general rule of thumb.
• Prefer larger pages to more requests. If the API accepts limit=100, one request for 100 records beats five for 20 — but test the ceiling, since many backends silently clamp an oversized limit back to their default and you will not notice the extra pages.

Common Errors and Fixes

403 Forbidden on a request that works in the browser. You are missing a header the backend checks. Add Referer, Origin, and X-Requested-With, and confirm the User-Agent matches a current browser build. If it still fails while cURL succeeds with the same headers, the discriminator is the TLS fingerprint, not HTTP — see the anti-bot material under Advanced Scraping Techniques and Anti-Bot Evasion.

401 Unauthorized partway through a long run. The bearer token expired. Read the exp claim and refresh pre-emptively as shown above, rather than discovering it through a failed request.

json.JSONDecodeError: Expecting value: line 1 column 1 (char 0). The response was not JSON — usually an HTML block page or a redirect to a login form. Log response.text[:500] in the handler so the next occurrence is diagnosable in one look.

httpx.ReadTimeout in a burst. The endpoint is throttling you. A burst of timeouts arriving together, rather than scattered, is the signature of rate limiting rather than a slow server; reduce concurrency before raising the timeout.

httpx.RemoteProtocolError: Server disconnected without sending a response. Often an HTTP/2 flow-control issue with an intermediary. Retry once, and if it persists on that host, construct the client with http2=False for it.

Empty items on page 1 when the browser shows results. The pagination or filter parameter name is wrong, and the server is falling back to a default that matches nothing. Re-check the exact query string in the Network tab — page versus p versus offset, pageSize versus limit — and match it character for character, including casing.

httpx.ConnectError: [SSL: CERTIFICATE_VERIFY_FAILED] behind a corporate proxy. The proxy is re-signing TLS with its own root. Point httpx at the corporate CA bundle with verify="/path/to/ca.pem" rather than disabling verification — turning verification off also hides the case where the endpoint itself has moved to a different host.

Results differ from the browser even though the request matches. The endpoint is personalising on something you did not reproduce: a geolocation cookie, an A/B test assignment, a currency preference, or a Accept-Language header. Compare the two response bodies field by field rather than assuming the request is wrong, since the request may be fine and the account context different.

The same page returned over and over. The cursor or page parameter is being ignored, usually because the API expects it in a POST body rather than the query string. Compare your request against the captured one field by field.

Frequently Asked Questions

Is calling a private API legal? This guide is technical, not legal advice. Requesting an undocumented endpoint is the same HTTP a browser makes, but you should review the site's terms of service and the law that applies to you, and avoid accessing data you are not authorised to see. Stay within what the frontend itself exposes to the account you are using.

Why bother with the API instead of parsing HTML? Speed, stability and completeness. A JSON endpoint returns structured data in one request with no browser to run and no selectors to maintain, and it frequently exposes fields the page never renders. When an internal API exists, it is almost always the better target.

How do I find the right request among hundreds in the Network tab? Filter to Fetch/XHR, clear the log, then trigger only the action you care about so the panel shows just the requests it caused. If several JSON responses remain, use the panel's search to grep across response bodies for a value you can see on the page — the request containing it is yours.

What if the endpoint needs a signature or hashed parameter? Some APIs sign requests with a value computed in JavaScript. You either read the site's bundle and port the signing logic to Python, or drive a real browser that computes it for you. Before doing either, test whether the server actually validates the signature: sending a stale or altered one sometimes works fine.

How often will a private endpoint break? Most changes are additive and harmless. Over a year of monitoring, the common breaking changes are a renamed pagination argument, a newly required header, and a path version bump. A contract assertion on the fields you read, run on every crawl, converts all three from silent data loss into a visible failure.

Related

• Data Extraction Patterns and APIs — the parent section and the other extraction routes
• Finding Hidden API Endpoints in Network Traffic — the discovery workflow in detail
• Scraping GraphQL Endpoints — when the site exposes one POST URL instead of many
• Scraping Mobile App APIs — the same idea applied to phone traffic
• Extracting JSON-LD and Structured Data — the fallback when no API is reachable