Finding Hidden API Endpoints in Network Traffic

This is the hands-on companion to Reverse-Engineering Private APIs in Python, focused on the one skill that unlocks everything else: spotting the hidden JSON endpoint inside a flood of network requests.

Open DevTools, switch to the Network panel, and filter to Fetch/XHR. Clear the log, then trigger the action that loads your data — scroll, click, search — so only the relevant requests appear. Click each one and read the Response tab until you find your JSON, then right-click and Copy as cURL to capture the exact URL, method, headers and body. For traffic DevTools cannot see, such as a mobile app or a service worker's calls, put mitmproxy in the middle and watch the same requests there.

Why Filtering Beats Scanning

A content-heavy page issues 200–400 requests on first load: images, fonts, stylesheets, analytics beacons, chunk after chunk of JavaScript. Reading that list top to bottom is hopeless. Two filters cut it to something a person can inspect.

The Fetch/XHR filter removes every subresource the browser fetched for rendering and leaves only the calls JavaScript made deliberately. Clearing the log and repeating one action removes everything that fired during page load, leaving just the requests your action caused. From there, three habits find the endpoint in seconds:

• Search inside responses. The magnifying-glass search in the Network panel greps across all response bodies, not just URLs. Paste a value you can see on the page — a price, a product name, an internal ID — and the request containing it is your endpoint. This is the single highest-yield technique on the list and the one most people never discover.
• Sort by size. Data responses are usually the largest Fetch/XHR entries. Sorting by the Size column floats them to the top and pushes empty beacons to the bottom.
• Ignore OPTIONS. Those are CORS preflights, not data calls. The GET or POST that immediately follows is the real request.

Once you click a candidate, the detail pane shows Headers (URL, method, request headers), Payload (query string or POST body) and Response (the raw JSON). If the Response tab holds what you want, you have found it. Everything after this point is reproducing that request faithfully, which is the subject of Understanding HTTP Requests and Responses.

Copy as cURL, Then Reduce

Reproducing a captured request header by header is error-prone. Right-click the request and choose Copy → Copy as cURL; that command captures the method, URL, every header, and the body verbatim. A captured command looks like this:

curl 'https://www.example.com/api/v2/listings?page=1&sort=price_asc' \
  -H 'accept: application/json' \
  -H 'accept-language: en-GB,en;q=0.9' \
  -H 'referer: https://www.example.com/listings' \
  -H 'sec-fetch-mode: cors' \
  -H 'sec-fetch-site: same-origin' \
  -H 'user-agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36' \
  -H 'x-requested-with: XMLHttpRequest'

The instinct is to copy all of that into Python. Resist it: a request with twenty headers has twenty things that can silently stop mattering, and you will never know which three were load-bearing. Reduce instead.

python -m pip install "httpx[http2]>=0.27"

import httpx

UA = ("Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 "
      "(KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36")

# Everything the browser sent, in browser order.
CAPTURED = {
    "accept": "application/json",
    "accept-language": "en-GB,en;q=0.9",
    "referer": "https://www.example.com/listings",
    "sec-fetch-mode": "cors",
    "sec-fetch-site": "same-origin",
    "user-agent": UA,
    "x-requested-with": "XMLHttpRequest",
}
# The three you start from.
MINIMAL = {"accept": "application/json", "referer": CAPTURED["referer"], "user-agent": UA}


def try_headers(url: str, headers: dict[str, str]) -> tuple[int, int]:
    with httpx.Client(http2=True, timeout=15) as client:
        response = client.get(url, headers=headers)
        return response.status_code, len(response.content)


def find_required_headers(url: str) -> dict[str, str]:
    """Start minimal; add one captured header at a time until the call succeeds."""
    headers = dict(MINIMAL)
    status, size = try_headers(url, headers)
    for name, value in CAPTURED.items():
        if status == 200 and size > 200:
            break
        if name in headers:
            continue
        headers[name] = value
        status, size = try_headers(url, headers)
    if status != 200:
        raise RuntimeError(f"still failing with {len(headers)} headers: HTTP {status}")
    return headers


if __name__ == "__main__":
    required = find_required_headers("https://www.example.com/api/v2/listings?page=1")
    print(f"minimal working header set: {sorted(required)}")

Two details make this work in practice. The size > 200 check catches the case where a block page returns HTTP 200 with a tiny body — a status check alone would declare victory. And keeping the header keys lowercase and in browser order matters because some anti-bot layers compare the ordering of a client's headers against known browser profiles; httpx preserves insertion order, so mirroring the capture costs nothing.

When DevTools Cannot See the Traffic

DevTools only observes the browser tab it is attached to. Mobile apps, desktop clients, Electron apps and some service-worker-mediated calls stay invisible. mitmproxy sits between the client and the internet as an intercepting HTTPS proxy, so every request flows through a log you control.

python -m pip install "mitmproxy>=10.0"

Start the interactive UI with mitmproxy, point the device or browser at the proxy on port 8080, install mitmproxy's CA certificate so TLS can be decrypted, and every request appears in the flow list. To capture programmatically rather than watching by hand, run an addon:

# save as capture.py, run with:  mitmdump -s capture.py -w flows.mitm
import json
from mitmproxy import http

INTERESTING = ("/api/", "/graphql", "/v1/", "/v2/", "/rest/")


def response(flow: http.HTTPFlow) -> None:
    url = flow.request.pretty_url
    content_type = flow.response.headers.get("content-type", "")
    if "application/json" not in content_type:
        return
    if not any(part in url for part in INTERESTING):
        return
    body = flow.response.content or b""
    print(f"[{flow.request.method}] {flow.response.status_code} {len(body):>7} B  {url}")
    if flow.request.method == "POST" and flow.request.content:
        print(f"    body: {flow.request.get_text()[:200]}")
    try:
        top = json.loads(body)
        keys = sorted(top)[:8] if isinstance(top, dict) else ["<list>"]
        print(f"    keys: {keys}")
    except ValueError:
        pass

Printing the top-level keys of each response turns the log into a map of the interface: you can see at a glance which endpoint returns {"items": ...} and which returns {"data": {...}}. The -w flows.mitm flag writes a replayable capture file, so you can re-analyse a session offline instead of repeating the interaction on the device. Applying this to a phone app end to end — proxy setup, certificate trust, and the pinning problem — is covered in Intercepting App Traffic with mitmproxy. For GraphQL traffic, the same capture surfaces the /graphql POST body you then replay following Scraping GraphQL Endpoints.

Analysing a Whole Session Offline With a HAR File

Clicking through requests one at a time is fine for a single endpoint and painful for mapping a whole application. The Network panel can export everything it logged as a HAR file — right-click the request list and choose "Save all as HAR with content" — which is plain JSON you can analyse in Python.

import json
from collections import Counter
from urllib.parse import urlsplit


def summarise_har(path: str, min_bytes: int = 500) -> None:
    with open(path, encoding="utf-8") as handle:
        har = json.load(handle)
    hosts: Counter[str] = Counter()
    for entry in har["log"]["entries"]:
        request, response = entry["request"], entry["response"]
        mime = response.get("content", {}).get("mimeType", "")
        size = response.get("content", {}).get("size", 0)
        if "json" not in mime or size < min_bytes:
            continue
        parts = urlsplit(request["url"])
        hosts[parts.netloc] += 1
        print(f"{request['method']:5} {response['status']} {size:>8} B  "
              f"{parts.netloc}{parts.path}")
    print("\nJSON endpoints per host:")
    for host, count in hosts.most_common():
        print(f"  {count:>4}  {host}")

Two things make this worth the extra step. The per-host tally immediately separates the site's own API from third-party analytics and ad calls, which is otherwise a manual read of every row. And because the HAR captures response bodies, you can grep the file for a value from the page without re-running the interaction — useful when the action that loads the data is hard to reproduce, such as a checkout step or a one-time onboarding flow. Treat the file as sensitive: "with content" means it contains every cookie, token and response body from the session.

Edge Cases and Caveats

• WebSocket data. If the panel shows a ws:// or wss:// connection carrying your data, it is a stream, not a request/response call. Inspect the Messages tab; you will need a WebSocket client such as websockets rather than httpx, and you may have to replay a subscription frame to start the flow.
• Certificate pinning. Some mobile apps refuse mitmproxy's CA and drop the connection with a TLS error rather than a useful message. Pinning blocks interception on unrooted devices and usually means the app actively resists inspection — see Scraping Mobile App APIs.
• Signed or hashed parameters. A query parameter that looks like a random hash is computed in JavaScript. You must read the site's bundle to reproduce it, or drive a browser that computes it for you. Test first whether the server actually validates it.
• Endpoints behind anti-bot layers. A request that works from the browser but returns 403 from Python with identical headers is being fingerprinted below the HTTP layer, at the TLS handshake — the subject of TLS and JA3 Fingerprint Evasion.
• Data already in the HTML. Before hunting for an endpoint, search the initial document for a value from the page. Server-rendered frameworks often ship the whole dataset in a __NEXT_DATA__ script tag, and reading that is simpler than any API call.
• Requests that only fire once. Some data loads on first visit and is then cached in localStorage. Use a private window, or disable cache in the Network panel, so the call happens again while you are watching.
• Server-side rendering means there is no endpoint to find. Frameworks that stream HTML from the server may never make a client-side data call at all, and the payload arrives inside the document as a script tag. An empty Fetch/XHR list on a data-rich page is evidence of this, not of failure to look properly.
• The response is compressed or chunked. DevTools shows the decoded body while the Size column shows transferred bytes, so a large response can look small. Compare the "transferred" and "resources" figures if a candidate looks too small to hold your data.

Frequently Asked Questions

The Network tab shows hundreds of requests. How do I narrow it down? Filter to Fetch/XHR, clear the log, then trigger only the action you care about so the panel shows just the requests it caused. If several JSON requests remain, use the panel's search to grep across response bodies for a value visible on the page — the request containing it is the one you want.

What is the difference between the Fetch/XHR filter and the Doc filter? "Doc" shows the top-level HTML document the browser first loaded, which is where server-rendered data lives. Fetch/XHR shows the background data calls JavaScript made afterwards. Private APIs almost always appear under Fetch/XHR, but check Doc first in case the data was never fetched separately at all.

Do I need mitmproxy if I only scrape websites? Usually not, because DevTools sees all browser traffic. Reach for mitmproxy when you target a mobile or desktop client, when a service worker intercepts calls before DevTools logs them cleanly, or when you want a replayable capture file of a whole session rather than a live panel.

Copy as cURL gave me a huge header list. Do I need all of it? No. Start with User-Agent, Accept and Referer, then add captured headers back one at a time until the request succeeds. Once it does, stop — the smaller set is the one you can maintain, and you now know exactly which headers the server actually checks.

Related

• Reverse-Engineering Private APIs in Python — the parent topic, covering replay, auth and pagination
• Scraping GraphQL Endpoints — what to do when the captured call is a GraphQL POST
• Parsing JSON and XML Responses — turning the captured payload into records
• Advanced Scraping Techniques and Anti-Bot Evasion — when a faithful replay still returns 403