Data Extraction Patterns and APIs

Once you can parse rendered HTML, a harder truth sets in: the cleanest copy of the data you want is almost never in the HTML. Modern pages hydrate themselves from JSON payloads, embed machine-readable application/ld+json blocks for search engines, and answer background fetch calls with tidy typed objects. Scraping the rendered markup means fighting CSS-class churn and layout changes; reading the data source directly means one stable request and a dictionary you can trust.

This path is for developers who have a working HTML scraper and are tired of repairing it. It sits between The Complete Guide to Python Web Scraping and Scaling Python Web Scrapers: everything here reduces both the fragility and the cost of a crawl, which is why it is worth doing before you scale anything. The payoff is concrete — a private API that returns 50 typed records per request replaces 50 page fetches and 50 selector expressions.

Stop Scraping the Page, Start Reading the Source

A rendered HTML element is a presentation of data, not the data itself. When a site restyles its product grid, your div.product-card__price--v2 selector breaks even though nothing about the underlying price changed. The fix is to move one layer down the stack. The same page usually carries the same values as JSON-LD, ships them in a JavaScript state blob, or fetches them from an endpoint that returns clean JSON.

The workflow you already know — fetch with a realistic User-Agent, then parse — still applies. What changes is what you parse. Compare a fragile selector against the structured alternative:

import requests
from bs4 import BeautifulSoup

HEADERS = {"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36"}

def price_from_markup(url: str) -> str | None:
    resp = requests.get(url, headers=HEADERS, timeout=10)
    resp.raise_for_status()
    soup = BeautifulSoup(resp.text, "lxml")
    tag = soup.select_one("span.price_color")   # breaks on any redesign
    return tag.get_text(strip=True) if tag else None

print(price_from_markup("https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html"))

That approach is covered end to end in Parsing HTML with BeautifulSoup. It is the right tool when the data genuinely only exists as visible text. The rest of this path is about the far more common case where it does not.

There is a second, less obvious argument for moving down the stack: cost. A rendered page transfers images, fonts, stylesheets, and analytics scripts alongside the twelve values you wanted. The equivalent API call transfers a few kilobytes of JSON. On a crawl of any size that difference shows up as bandwidth, as parsing time, and as load you are placing on someone else's servers.

Structured Data Is Already in the Page

Most commercial pages ship a block of structured data specifically so that Google, Bing, and social crawlers can read them. That block is a gift: it is a JSON object with a documented schema.org vocabulary, sitting in the HTML you already downloaded, and it changes far less often than the visible layout. Finding it is a single selector; parsing it is a single json.loads.

The stability argument is worth stating plainly. A site's marketing team can rebuild the entire product page and nothing about the ld+json block changes, because breaking it would cost them rich results in search — which is a revenue-visible consequence in a way that breaking your scraper is not. That incentive alignment is why JSON-LD outlives selectors.

import json
import requests
from bs4 import BeautifulSoup

HEADERS = {
    "User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 "
                  "(KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36",
}

def read_structured_data(url: str) -> list[dict]:
    """Every ld+json block, flattened: a block may be an object, a list, or a @graph."""
    resp = requests.get(url, headers=HEADERS, timeout=10)
    resp.raise_for_status()
    soup = BeautifulSoup(resp.text, "lxml")
    out: list[dict] = []
    for block in soup.find_all("script", type="application/ld+json"):
        if not block.string:
            continue
        try:
            payload = json.loads(block.string)
        except json.JSONDecodeError:
            continue                       # trailing commas and stray HTML happen
        items = payload if isinstance(payload, list) else [payload]
        for item in items:
            out.extend(item["@graph"] if isinstance(item, dict) and "@graph" in item else [item])
    return out

for node in read_structured_data("https://www.python.org/"):
    print(node.get("@type"), "->", list(node)[:6])

Three defensive details there are not optional. Blocks fail to parse more often than you expect, so the try is load-bearing. A block can be a bare object or an array. And a @graph key holds a list of nodes that must be unwrapped, which is how most WordPress and Yoast-generated markup is shaped. The full treatment of these blocks, plus microdata and Open Graph tags, lives in Extracting JSON-LD and Structured Data, with the commerce case in Scraping Schema.org Product Data.

Parsing JSON and XML the Server Hands You

When a request returns JSON or XML directly — a REST endpoint, an RSS feed, a sitemap, a data export — there is no markup to parse at all. The job becomes navigating a nested structure and pulling out the fields you need. Python's standard library reads JSON natively, and small helpers like xmltodict collapse XML into the same dict-and-list shape.

The trap in JSON work is not parsing; it is the shape of what you get. Optional fields disappear rather than arriving as null, nesting depth varies between records, and a field that is a list on one record is a bare string on another. Reaching in with payload["data"]["items"][0]["price"]["amount"] produces a KeyError or TypeError in the middle of a long run. Traverse defensively and decide once what a missing value means.

XML brings a different trap: namespaces. An element that displays as <url> in the browser is really {http://www.sitemaps.org/schemas/sitemap/0.9}url to ElementTree, and a findall("url") that ignores that returns nothing at all — silently.

import requests
from xml.etree import ElementTree

HEADERS = {
    "User-Agent": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 "
                  "(KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36",
    "Accept": "application/xml,text/xml;q=0.9,*/*;q=0.8",
}
NS = {"sm": "http://www.sitemaps.org/schemas/sitemap/0.9"}

def dig(payload: object, *path: str | int, default: object = None) -> object:
    """Walk a nested payload without raising on a missing key or index."""
    current = payload
    for step in path:
        try:
            current = current[step]          # type: ignore[index]
        except (KeyError, IndexError, TypeError):
            return default
    return current

def sitemap_urls(url: str, limit: int = 5) -> list[str]:
    resp = requests.get(url, headers=HEADERS, timeout=15)
    resp.raise_for_status()
    root = ElementTree.fromstring(resp.content)   # .content, not .text: keep the XML decl
    return [node.text or "" for node in root.findall(".//sm:loc", NS)][:limit]

print(dig({"data": {"items": [{"price": {"amount": "19.99"}}]}}, "data", "items", 0, "price", "amount"))
print(dig({"data": {}}, "data", "items", 0, "price", default="missing"))
print(sitemap_urls("https://www.python.org/sitemap.xml"))

Passing resp.content rather than resp.text to ElementTree.fromstring matters: the XML declaration names the document's encoding, and handing the parser an already-decoded string makes that declaration a lie it may act on. For JSONPath queries, streaming large payloads, and per-format tooling see Parsing JSON and XML Responses, the nested-to-tabular step in Flattening Nested JSON with pandas, and crawl-planning from sitemaps in Parsing XML Sitemaps with Python.

Reverse-Engineering the Private API Behind a Page

The most valuable endpoints are the undocumented ones the site's own front end calls. When you scroll a listing or open a product, the browser fires an XHR or fetch request that returns exactly the data the UI renders — paginated, typed, and free of markup. Replaying that request in Python is faster and sturdier than driving a browser or parsing HTML, because you are talking to the same interface the app does.

The method is mechanical. Open DevTools, filter the Network panel to Fetch/XHR, interact with the page, and look for a response whose JSON contains a value you can see on screen. Right-click it and copy as cURL — that gives you the exact request including every header and cookie. Then trim: paste it into Python, confirm it still works, and remove headers one at a time until it breaks. What remains is the minimum contract, and it is usually four or five headers rather than the twenty-three the browser sent.

import requests

SESSION_HEADERS = {
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 "
                  "(KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36",
    "Accept": "application/json",
    "X-Requested-With": "XMLHttpRequest",
    "Referer": "https://httpbin.org/",
}

def replay(endpoint: str, page: int) -> dict:
    """The trimmed request: only the headers the endpoint actually checks."""
    response = requests.get(
        endpoint,
        params={"page": page, "limit": 50},
        headers=SESSION_HEADERS,
        timeout=15,
    )
    response.raise_for_status()
    if "json" not in response.headers.get("Content-Type", ""):
        raise RuntimeError(f"expected JSON, got {response.headers.get('Content-Type')}")
    return response.json()

echo = replay("https://httpbin.org/get", page=1)
print(echo["args"], echo["headers"]["X-Requested-With"])

Authentication is the one part of this that will surprise you later rather than immediately. Many internal endpoints accept a short-lived bearer token or a signed query parameter whose lifetime is measured in minutes, so a request that works in the terminal returns 401 the following morning. The fix is never to paste a captured token into your code: find the call the front end makes at page load to obtain it, replay that first, and derive a fresh token at the start of every run.

The Content-Type assertion is the guard that turns a confusing failure into a clear one. When an endpoint decides you are unauthenticated it frequently returns an HTML login page with a 200 status, and response.json() then raises a JSONDecodeError that points at parsing rather than at authentication. The systematic method is in Reverse-Engineering Private APIs, and the discovery step specifically in Finding Hidden API Endpoints in Network Traffic.

Reading the Response Envelope and Paging It

An API response is rarely a bare list. It is an envelope with three functional parts, and knowing which is which turns a one-page fetch into a complete extraction.

The records live under a key such as results, items, data, or edges. Alongside them sits metadata — a total, a page number, a per-page size — which is the cheapest correctness check you will ever get: if the total says 4,821 and you collected 4,300, you know to look for the gap rather than shipping the dataset. And somewhere there is the paging control, either an offset you increment or an opaque cursor you must echo back.

The distinction matters operationally. Offset paging is parallelisable, because page 7 can be requested without knowing page 6. Cursor paging is strictly sequential, because the token for the next call only exists in the previous response — which means it cannot be spread across workers and needs its own resume strategy.

import time
import requests

HEADERS = {
    "User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 "
                  "(KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36",
    "Accept": "application/json",
}

def page_by_cursor(endpoint: str, max_pages: int = 20) -> list[dict]:
    """Sequential cursor paging with a hard stop and a stall guard."""
    records: list[dict] = []
    cursor: str | None = None
    seen_cursors: set[str] = set()
    for _ in range(max_pages):
        params: dict[str, object] = {"limit": 50}
        if cursor:
            params["cursor"] = cursor
        response = requests.get(endpoint, params=params, headers=HEADERS, timeout=15)
        response.raise_for_status()
        payload = response.json()

        batch = payload.get("results") or []
        records.extend(batch)
        cursor = (payload.get("page_info") or {}).get("next_cursor")
        if not cursor or cursor in seen_cursors or not batch:
            break                                  # end of data, or the server is looping
        seen_cursors.add(cursor)
        time.sleep(0.5)
    return records

print(len(page_by_cursor("https://httpbin.org/json")), "records")

The seen_cursors set is not paranoia. Endpoints that return a stale or constant cursor at the end of a dataset are common enough that a loop without this guard will run until it hits max_pages every time. Cursor semantics in the GraphQL flavour are covered in Handling GraphQL Pagination and Cursors.

Querying GraphQL Endpoints Directly

A growing number of sites expose a single GraphQL endpoint instead of many REST routes. That looks intimidating but is often easier to scrape: you send a POST with a query describing exactly the fields you want, and the server returns exactly those and nothing else. No over-fetching, no scraping around unrelated markup.

Two properties make GraphQL pleasant for extraction. Field selection means you can ask for three fields instead of downloading a 40 KB object to read three fields. And the schema is typed, so a field either exists with a known type or the request fails loudly at the top level rather than silently producing None deep in your parsing code. The wrinkle is error handling: a GraphQL server returns 200 OK with an errors array for a failed query, so raise_for_status() alone will happily hand you an empty result.

import requests

HEADERS = {
    "User-Agent": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 "
                  "(KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36",
    "Content-Type": "application/json",
    "Accept": "application/json",
}

QUERY = """
query Products($first: Int!, $after: String) {
  products(first: $first, after: $after) {
    pageInfo { hasNextPage endCursor }
    edges { node { id name priceRange { minVariantPrice { amount currencyCode } } } }
  }
}
"""

def graphql(endpoint: str, query: str, variables: dict) -> dict:
    response = requests.post(
        endpoint,
        json={"query": query, "variables": variables},
        headers=HEADERS,
        timeout=20,
    )
    response.raise_for_status()
    payload = response.json()
    if payload.get("errors"):
        messages = "; ".join(e.get("message", "?") for e in payload["errors"])
        raise RuntimeError(f"GraphQL error: {messages}")   # a 200 that is really a failure
    return payload["data"]

print(graphql.__doc__ or "posts a query and raises on the errors array")
print(QUERY.strip().splitlines()[0])

Checking the errors array before touching data is the difference between a clear exception and a scraper that quietly writes zero rows for a week. Query construction, schema introspection, and cursor paging are covered in Scraping GraphQL Endpoints.

Cleaning and Validating What Comes Back

Structured sources are cleaner than markup, not clean. JSON-LD routinely stringifies numbers ("price": "19.99"), private APIs return dates in three formats across two endpoints, currency symbols travel inside the value, and the same product appears twice with a trailing space in its title. If those pass through untouched, the problem has been moved downstream rather than solved.

The discipline is to validate at the boundary — the moment a payload enters your code — and to count what fails. A model that coerces types, normalises units, and rejects the impossible gives you two things: records you can compute on, and a reject rate that acts as an early-warning signal when the source changes shape. Full coverage is in Cleaning and Validating Scraped Data, with the schema layer in Validating Scraped Data with Pydantic, unit and format handling in Normalizing Prices, Dates and Units, and near-duplicate handling in Deduplicating Records with Fuzzy Matching.

# pip install "pydantic==2.9.2"
import re
from datetime import date, datetime
from pydantic import BaseModel, Field, ValidationError, field_validator

class Product(BaseModel):
    sku: str = Field(min_length=1)
    name: str
    price: float = Field(gt=0)
    currency: str = Field(min_length=3, max_length=3)
    listed_on: date

    @field_validator("price", mode="before")
    @classmethod
    def as_number(cls, value: str | float) -> float:
        if isinstance(value, str):
            return float(re.sub(r"[^\d.]", "", value))
        return value

    @field_validator("listed_on", mode="before")
    @classmethod
    def as_date(cls, value: str | date) -> date:
        if isinstance(value, date):
            return value
        for fmt in ("%Y-%m-%d", "%d/%m/%Y", "%d %B %Y"):
            try:
                return datetime.strptime(value, fmt).date()
            except ValueError:
                continue
        raise ValueError(f"unrecognised date: {value!r}")

    @field_validator("name")
    @classmethod
    def tidy(cls, value: str) -> str:
        return re.sub(r"\s+", " ", value).strip()

def ingest(rows: list[dict]) -> tuple[list[Product], list[str]]:
    good: list[Product] = []
    errors: list[str] = []
    for row in rows:
        try:
            good.append(Product(**row))
        except ValidationError as exc:
            errors.append(f"{row.get('sku', '?')}: {exc.error_count()} problem(s)")
    return good, errors

payload = [
    {"sku": "A1", "name": "  Wide   Lamp ", "price": "£19.99", "currency": "GBP",
     "listed_on": "3 May 2026"},
    {"sku": "A2", "name": "Broken", "price": "free", "currency": "GBP",
     "listed_on": "2026-05-03"},
]
products, failures = ingest(payload)
print(products[0].model_dump())
print("rejected:", failures)

The first record is coerced into a float, a real date, and a whitespace-normalised name; the second is rejected with a countable error rather than silently stored as garbage. Tracking that reject count per run is the single most effective way to notice that an upstream source changed.

Choosing the Right Source for a Page

Given a target page, work down this ladder and stop at the first source that has your data cleanly:

1. A private JSON API — the cleanest, most stable option when it exists, and usually the fastest to discover.
2. A GraphQL endpoint — nearly as clean; you control the field selection and pay only for what you ask for.
3. JSON-LD structured data — no network archaeology needed; it is in the HTML you already have, and it survives redesigns.
4. Rendered HTML — the fallback for data that genuinely exists nowhere else.

The trade-off is discovery effort against durability. Parsing HTML costs zero discovery and is the most fragile; a private API costs ten minutes in the Network panel and rarely breaks. On any non-trivial project the API route pays for itself within the first schema change on the target. The one case that inverts the ranking is a page where the API is authenticated and the HTML is not — then the markup is genuinely the lower-friction source.

def choose_strategy(has_api: bool, has_graphql: bool, has_jsonld: bool) -> str:
    """The ladder as code: first source that holds the data wins."""
    if has_api:
        return "replay the private JSON request"
    if has_graphql:
        return "post a field-scoped GraphQL query"
    if has_jsonld:
        return "parse the ld+json block"
    return "fall back to HTML selectors"

for flags in [(True, False, True), (False, True, True), (False, False, True), (False, False, False)]:
    print(flags, "->", choose_strategy(*flags))

Storing What You Extract

Every source above converges on the same output: Python dicts and lists, validated, ready to persist. Because API and JSON-LD data arrives already typed and nested, the natural next step is to flatten and store it, and the format choice follows the same rules as any other crawl — append as you go, key on something stable, and make a re-run idempotent.

import csv
import json

def rows_to_jsonl(rows: list[dict], path: str) -> int:
    """Append-as-you-go: a crash costs the last record, not the run."""
    with open(path, "a", encoding="utf-8") as handle:
        for row in rows:
            handle.write(json.dumps(row, ensure_ascii=False, sort_keys=True) + "\n")
    return len(rows)

def rows_to_csv(rows: list[dict], path: str) -> None:
    if not rows:
        return
    fieldnames = sorted({key for row in rows for key in row})   # union, not row 0
    with open(path, "w", newline="", encoding="utf-8") as handle:
        writer = csv.DictWriter(handle, fieldnames=fieldnames, restval="")
        writer.writeheader()
        writer.writerows(rows)

records = [{"sku": "A1", "price": 19.99}, {"sku": "A2", "price": 4.5, "currency": "GBP"}]
print(rows_to_jsonl(records, "items.jsonl"), "appended")
rows_to_csv(records, "items.csv")

Taking the union of keys rather than list(rows[0].keys()) is the fix for the most common CSV bug in scraping code: the first record lacks an optional field, so the column never appears in the header and every later value for it is silently dropped. The hand-off to databases and columnar formats is in Storing and Exporting Scraped Data.

Common Pitfalls

Structured sources fail differently from HTML. An HTML scraper breaks loudly, with an empty result set you notice; a payload-based scraper tends to break quietly, storing wrong types or half a dataset while reporting success. The list below is ordered by how often each one is the real cause.

• Rendering the page when the JSON was right there. Reaching for Selenium or Playwright before checking the Network panel wastes CPU and time. Look for a JSON source first; render only when there truly is none.
• Dropping headers on API calls. Private endpoints often gate on Accept, Referer, X-Requested-With, or a token header. Copy the request the browser actually sent, headers and all, then trim one at a time.
• Assuming JSON-LD is always one object. A page can contain several ld+json blocks, and each may be a single object, an array, or a @graph list. Iterate defensively and tolerate a block that fails to parse.
• Ignoring pagination shape. REST offsets, cursor tokens, and GraphQL edges/pageInfo cursors all page differently, and only offsets parallelise. Read the envelope before writing the loop.
• Hammering an undocumented endpoint. A private API has no published rate limit, which means it has an unknown one. Throttle and back off exactly as you would for HTML requests.
• Trusting types blindly. JSON-LD often stringifies numbers and dates arrive in whatever format the template used. Cast and validate at the boundary before doing arithmetic or comparisons.
• Treating a 200 as success on GraphQL. A failed GraphQL query returns 200 with an errors array and a null data. Check errors explicitly or you will store nothing and never know.
• Parsing XML without namespaces. findall("url") on a namespaced document returns an empty list rather than an error, which reads exactly like an empty feed.
• Hard-coding a captured token. Bearer tokens and signed query parameters copied out of DevTools expire, often within minutes. Fetch the token the way the front end does, at the start of every run.

Frequently Asked Questions

How do I know whether a site has a private API to scrape? Open the browser DevTools Network panel, filter to Fetch/XHR, and interact with the page — scroll, paginate, open a detail view. Any request that returns JSON matching what you see on screen is a candidate endpoint you can replay directly in Python. If the page is server-rendered you will see no such requests, which is itself a useful answer.

Is calling a site's private API legal? Reading a publicly reachable endpoint is technically no different from loading the page that calls it, but the same considerations apply as for any scraping: respect the terms of service, do not bypass authentication you were not granted, honour robots.txt, and throttle politely. This site does not give legal advice; treat undocumented endpoints as a convenience, not a licence.

Why prefer JSON-LD over parsing the visible HTML? JSON-LD is published as machine-readable structured data with a documented schema.org vocabulary, so it is both cleaner and far more stable than presentation markup. A site can redesign its entire product grid without touching the ld+json block, because breaking that block would cost it search visibility.

Do I still need BeautifulSoup if I am reading APIs? Often yes — to locate the <script type="application/ld+json"> blocks inside a page, and as a fallback for values that only exist in visible markup. For pure JSON or XML endpoints you can skip it entirely and parse the response body directly.

What is the difference between a REST API and a GraphQL endpoint for scraping? A REST API exposes many URLs, each returning a fixed shape, and you page through them with query parameters. A GraphQL endpoint is a single URL you POST queries to, choosing exactly which fields come back. GraphQL avoids over-fetching but requires you to write the query, handle cursor pagination, and check the errors array on every response.

Can I mix these techniques in one scraper? Yes, and mature scrapers usually do. A common pattern is to page a listing through a private JSON API, enrich each item with JSON-LD pulled from its detail page, validate the merged record, and store it — each source used where it is cleanest.

Related

• Parsing JSON and XML Responses — navigating nested payloads, namespaces, and large documents
• Extracting JSON-LD and Structured Data — ld+json, microdata, and Open Graph
• Reverse-Engineering Private APIs — finding, replaying, and authenticating against internal endpoints
• Scraping GraphQL Endpoints — queries, introspection, and cursor paging
• Cleaning and Validating Scraped Data — schemas, normalisation, and deduplication
• The Complete Guide to Python Web Scraping — the HTML fundamentals this path replaces where it can
• Advanced Scraping Techniques and Anti-Bot Evasion — what to do when the endpoint is defended
• Scaling Python Web Scrapers — running API-based extraction at volume