Parsing JSON and XML API Responses in Python

When an endpoint hands you JSON or XML directly, there is no HTML to fight — this guide sits inside Data Extraction Patterns and APIs because the work shifts entirely to navigating a nested structure and pulling out exactly the fields you need. It covers the everyday tools for both formats: the standard-library json module and JSONPath for JSON, and xmltodict and lxml for XML feeds, sitemaps, and legacy SOAP-style responses, along with the specific exceptions each one raises when the payload is not what you assumed.

When to Use Each Tool

Pick the parser by the format, the size of the document, and the depth of the query you need. The four options are not interchangeable, and choosing the wrong one usually shows up as either a memory spike or an unhandled exception on record 4,000 of a run.

A few decision rules that hold up in practice:

• json (standard library) is the default for any JSON body. requests calls it for you through response.json(). Reach past it only when navigation gets awkward, not because a third-party library sounds faster — CPython's json C extension decodes roughly 100–200 MB/s and is rarely the bottleneck in a network-bound scraper.
• JSONPath (jsonpath-ng) earns its place when you need values from deeply nested or variable JSON with a query expression instead of a chain of ["key"][0]["key"] lookups. Its real advantage is that a non-matching path returns an empty list rather than raising, which turns a class of crashes into a class of missing values you can count.
• xmltodict is the fastest route from XML to ordinary Python dicts and lists. It is built on expat and produces the same shape as parsed JSON, so a downstream pipeline can treat both identically.
• lxml is the right answer when XML is large, uses namespaces, or needs real XPath. It wraps libxml2, is several times faster than xml.etree.ElementTree on large documents, and is the only one of the four that can stream.

If you do not yet know whether an endpoint returns JSON or XML — or whether it exists at all — the discovery workflow lives in Reverse-Engineering Private APIs.

Prerequisites

Use Python 3.10 or newer, since the code below uses X | None union syntax in annotations. Install the HTTP client plus the two optional parsers:

python -m pip install "requests>=2.31" "xmltodict>=0.13" "lxml>=5.0" "jsonpath-ng>=1.6"

The json module ships with Python, so JSON parsing needs no install. lxml publishes manylinux, macOS and Windows wheels bundling libxml2, so no system package is required on any mainstream platform; if pip starts compiling from source you are on an unusual architecture and need libxml2-dev and libxslt1-dev. A refresher on how these responses arrive over the wire, including content negotiation and status handling, is in Understanding HTTP Requests and Responses.

Step-by-Step: From Response Bytes to Clean Records

1. Parse a JSON Response Safely

For well-behaved JSON APIs, response.json() is all you need. Send an explicit Accept header, check the status, then walk the parsed dict. The guard on 204 No Content matters more than it looks: a DELETE or an empty search result frequently answers with no body at all, and .json() on an empty body raises requests.exceptions.JSONDecodeError (a subclass of json.JSONDecodeError since requests 2.27).

import requests

HEADERS = {
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 "
                  "(KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36",
    "Accept": "application/json",
}


def fetch_json(url: str, params: dict[str, str | int] | None = None) -> dict:
    resp = requests.get(url, headers=HEADERS, params=params, timeout=10)
    resp.raise_for_status()
    if resp.status_code == 204 or not resp.content:
        return {}
    ctype = resp.headers.get("Content-Type", "")
    if "json" not in ctype:
        raise ValueError(f"expected JSON, got {ctype!r}: {resp.text[:120]!r}")
    return resp.json()


if __name__ == "__main__":
    payload = fetch_json("https://httpbin.org/json")
    print(payload["slideshow"]["title"])

Checking Content-Type before parsing converts the single most common production failure — an anti-bot interstitial or a login redirect returned with HTTP 200 — from a cryptic Expecting value: line 1 column 1 (char 0) into a message that tells you what actually arrived.

2. Query Deep JSON with JSONPath

When the value you want is buried several levels down, or its position varies between records, a JSONPath expression is far more readable than a long chain of subscripts and defends against missing keys by construction.

from jsonpath_ng.ext import parse
from jsonpath_ng.jsonpath import Fields

PRICE_EXPR = parse("$.results[*].variants[*].price")
CHEAP_EXPR = parse("$.results[*].variants[?(@.price < 20)].sku")


def extract_prices(payload: dict) -> list[float]:
    return [float(match.value) for match in PRICE_EXPR.find(payload)]


def cheap_skus(payload: dict) -> list[str]:
    return [match.value for match in CHEAP_EXPR.find(payload)]


def path_of_each_match(payload: dict) -> list[str]:
    return [str(match.full_path) for match in PRICE_EXPR.find(payload)]

Three details are worth internalising. First, compile the expression once at module level — parse() builds an AST, and re-parsing the same string inside a loop over 50,000 records is measurably slower than the traversal itself. Second, filter expressions like ?(@.price < 20) only exist in jsonpath_ng.ext, not in the base jsonpath_ng module; importing from the wrong one raises JsonPathParserError: Parse error at 1:22 near token ? (?). Third, match.full_path gives you the concrete path each value came from, which is invaluable when you need to write a value back or report which record was malformed.

Once you have flat values like these, turning a nested payload into rows for analysis is its own task — see Flattening Nested JSON with pandas, which covers record_path, meta, and the KeyError that appears when records disagree about which fields exist.

3. Parse XML with xmltodict

xmltodict collapses XML into the same dict-and-list shape as JSON, so the rest of your pipeline treats both formats identically. This is the pragmatic choice for RSS and Atom feeds, OPML, and small XML APIs where you would otherwise write ten lines of ElementTree ceremony.

import requests
import xmltodict

HEADERS = {
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 "
                  "(KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36",
    "Accept": "application/rss+xml, application/xml;q=0.9",
}


def as_list(value: object) -> list:
    """XML with exactly one child parses to a dict, not a one-element list."""
    if value is None:
        return []
    return value if isinstance(value, list) else [value]


def parse_rss(url: str) -> list[dict[str, str | None]]:
    resp = requests.get(url, headers=HEADERS, timeout=10)
    resp.raise_for_status()
    doc = xmltodict.parse(resp.content)
    items = as_list(doc["rss"]["channel"].get("item"))
    return [
        {
            "title": item.get("title"),
            "link": item.get("link"),
            "published": item.get("pubDate"),
            "guid": (item.get("guid") or {}).get("#text") if isinstance(item.get("guid"), dict) else item.get("guid"),
        }
        for item in items
    ]

The as_list helper is not optional. xmltodict has no schema, so it infers cardinality from the document instance: a channel with five <item> elements yields a list, a channel with one yields a bare dict, and iterating that dict gives you its keys rather than its records. Feeds that are usually busy and occasionally quiet fail this way once a quarter, which makes it hard to reproduce.

Two more xmltodict behaviours to know: attributes are prefixed with @ (item["@id"]), and mixed text-plus-attribute nodes become a dict with a #text key, which is why the guid extraction above tests the type first. Both prefixes are configurable through the attr_prefix and cdata_key arguments.

4. Parse Large or Namespaced XML with lxml

For big documents, XPath queries, or XML that declares namespaces, lxml is the right tool. Namespaces are the single most common reason an XPath that "looks correct" returns nothing.

import requests
from lxml import etree

HEADERS = {
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 "
                  "(KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36",
    "Accept": "application/xml",
}
SITEMAP_NS = {"sm": "http://www.sitemaps.org/schemas/sitemap/0.9"}


def parse_sitemap(url: str) -> list[dict[str, str | None]]:
    resp = requests.get(url, headers=HEADERS, timeout=20)
    resp.raise_for_status()
    root = etree.fromstring(resp.content)
    entries: list[dict[str, str | None]] = []
    for node in root.findall(".//sm:url", SITEMAP_NS):
        loc = node.find("sm:loc", SITEMAP_NS)
        lastmod = node.find("sm:lastmod", SITEMAP_NS)
        entries.append({
            "loc": loc.text if loc is not None else None,
            "lastmod": lastmod.text if lastmod is not None else None,
        })
    return entries


if __name__ == "__main__":
    for entry in parse_sitemap("https://www.python.org/sitemap.xml")[:5]:
        print(entry)

The SITEMAP_NS dictionary maps a short prefix to the namespace URI so .//sm:loc matches the namespaced <loc> elements. Internally libxml2 stores that element's name as {http://www.sitemaps.org/schemas/sitemap/0.9}loc, so a query for the bare name loc genuinely does not match anything — the failure is silent because an empty node set is a legal XPath result, not an error.

You can avoid the prefix map entirely by writing the Clark notation directly — root.findall(".//{http://www.sitemaps.org/schemas/sitemap/0.9}loc") — or, if you use root.xpath() rather than findall(), by matching on local name: root.xpath("//*[local-name()='loc']"). The local-name() trick is a reasonable escape hatch for one-off work but costs roughly 2–3× the evaluation time of a prefixed query on a large document, because it cannot use the name index. Sitemaps in particular have enough quirks — index files, gzip, changefreq fields, 50,000-URL limits — that they get their own treatment in Parsing XML Sitemaps with Python.

5. Stream Documents That Do Not Fit in Memory

etree.fromstring builds the entire tree. A 400 MB XML export becomes roughly 2–4 GB of C structs, which is how a scraper that worked on a sample kills a 2 GB container. iterparse handles the document as a stream and lets you discard each record after you have read it.

import requests
from lxml import etree

HEADERS = {
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 "
                  "(KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36",
}
PRODUCT_TAG = "{http://example.com/feed}product"


def stream_products(url: str):
    with requests.get(url, headers=HEADERS, stream=True, timeout=60) as resp:
        resp.raise_for_status()
        resp.raw.decode_content = True
        for _event, elem in etree.iterparse(resp.raw, events=("end",), tag=PRODUCT_TAG):
            yield {child.tag.rsplit("}", 1)[-1]: child.text for child in elem}
            elem.clear()
            while elem.getprevious() is not None:
                del elem.getparent()[0]


if __name__ == "__main__":
    for count, record in enumerate(stream_products("https://example.com/feed/products.xml"), 1):
        if count > 3:
            break
        print(record)

The three-line cleanup after yield is the part people omit. elem.clear() empties the element, but the parent still holds a reference to the now-empty node, so a document with two million records still accumulates two million shells. Deleting the preceding siblings keeps resident memory flat — typically under 30 MB regardless of file size. resp.raw.decode_content = True makes requests transparently gunzip a Content-Encoding: gzip stream; without it, iterparse receives compressed bytes and raises XMLSyntaxError: Start tag expected, '<' not found, line 1, column 1.

6. Read JSON That Arrives Inside HTML

Not every JSON payload comes back with a JSON content type. Server-rendered React and Next.js applications ship their state inside the page as <script id="__NEXT_DATA__" type="application/json">, Nuxt writes window.__NUXT__, and many templating stacks inline a window.__INITIAL_STATE__ = {...}; assignment. These are ordinary JSON documents once you cut them out of the surrounding markup, and reading them is usually faster and far more stable than selecting the same values from the rendered DOM.

import json
import re
import requests

HEADERS = {
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 "
                  "(KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36",
    "Accept": "text/html,application/xhtml+xml",
}
STATE_RE = re.compile(
    r"<script[^>]+id=\"__NEXT_DATA__\"[^>]*>(.*?)</script>", re.DOTALL
)


def embedded_state(url: str) -> dict:
    resp = requests.get(url, headers=HEADERS, timeout=15)
    resp.raise_for_status()
    match = STATE_RE.search(resp.text)
    if match is None:
        raise LookupError("no __NEXT_DATA__ block in the response")
    return json.loads(match.group(1))

Two failure modes recur here. A window.__INITIAL_STATE__ assignment ends with a semicolon that is not part of the JSON, so slicing to the last } before it is required or json.loads raises Extra data: line 1 column N. And a string value inside the payload can legally contain the characters </script>, which the server escapes as <\/script>; a naive regex that stops at the first closing tag it sees will truncate the document mid-object and produce Unterminated string starting at. The related case of purpose-published metadata blocks is covered in Extracting JSON-LD and Structured Data.

7. Hand Clean Records Downstream

Parsing is only the first half. API payloads carry stringified numbers, ISO timestamps in three different offsets, and null where you expected a list. Coerce and check the record at the boundary, before it reaches storage, so that a schema change shows up as one loud failure rather than a slow drift in your data:

from datetime import datetime, timezone


def coerce_product(raw: dict) -> dict[str, object]:
    price = raw.get("price")
    updated = raw.get("updated_at")
    return {
        "sku": (raw.get("sku") or "").strip().upper() or None,
        "price": float(str(price).replace(",", "")) if price not in (None, "") else None,
        "currency": (raw.get("currency") or "USD").upper(),
        "in_stock": bool(raw.get("stock", 0)),
        "updated_at": (
            datetime.fromisoformat(updated.replace("Z", "+00:00")).astimezone(timezone.utc)
            if updated else None
        ),
    }

The full set of coercion, validation and de-duplication patterns — including declarative models that reject a bad record instead of silently coercing it — is in Cleaning and Validating Scraped Data.

Performance and Scaling Considerations

• Stream, do not slurp, huge feeds. For XML measured in hundreds of megabytes use iterparse with the sibling-deletion pattern above. The difference is not marginal: a 1 GB catalogue is a few tens of MB resident when streamed and an out-of-memory kill when parsed whole.
• response.json() beats json.loads(response.text). The latter decodes bytes to str and then re-encodes internally; letting requests do it saves one full copy of the payload. On a 40 MB body that is 40 MB of avoidable allocation per request.
• Compile JSONPath expressions once. parse(...) builds an expression object. Reusing it across a loop of 100,000 records typically halves the time spent in the library compared with re-parsing the string each iteration.
• Prefer response.content over response.text for XML. Passing bytes lets the parser honour the document's own <?xml encoding=...?> declaration. Passing a str that contains an encoding declaration raises ValueError: Unicode strings with encoding declaration are not supported.
• Swap in orjson only when profiling says so. It is roughly 2–5× faster than stdlib json on decode, but it returns bytes from dumps and rejects some non-standard inputs stdlib accepts. In a scraper spending 95% of its wall time waiting on sockets, the win is invisible.
• Persist incrementally. Parsed dicts are storage-ready; write them in batches as described in Storing and Exporting Scraped Data rather than accumulating a list that grows for the length of the run.

Common Errors and Fixes

json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0) — the response was not JSON at all, usually an HTML error or login page returned with a 200. Check response.headers["Content-Type"] before parsing and log response.text[:200] in the exception handler, exactly as the fetch_json helper above does.

KeyError deep in a nested payload — a key is absent for some records. Replace chained subscripts with .get(...) defaults or a JSONPath query, both of which yield nothing instead of raising. When the missing key is genuinely required, raise your own error naming the record ID so the failure is diagnosable.

lxml.etree.XMLSyntaxError: Start tag expected or Extra content at the end of the document — the document is not well-formed, or you fed the parser a gzip stream. Pass resp.content (bytes), and for chronically messy vendor XML use a recovering parser: etree.fromstring(data, parser=etree.XMLParser(recover=True)), which skips broken nodes rather than aborting.

ValueError: Unicode strings with encoding declaration are not supported — you passed resp.text to etree.fromstring. Pass resp.content instead.

Namespaced XPath returns an empty list — you queried //loc on a document that namespaces its elements. Register the prefix map and query //sm:loc, or use Clark notation. Print root.nsmap to see exactly what the document declared; a None key in that dict is the default namespace and still applies to every child element.

TypeError: string indices must be integers when iterating xmltodict output — the element had exactly one child, so you are iterating a dict's keys. Normalise with the as_list helper before the loop.

requests.exceptions.JSONDecodeError on an empty body — the endpoint answered 204 No Content or returned zero bytes. Guard on resp.status_code == 204 or not resp.content before calling .json().

Numbers come back subtly wrong — a JSON document contained an integer larger than 2^53 or a decimal such as 0.1. Python's json maps every JSON number to int or float, so a 64-bit identifier survives (Python integers are arbitrary precision) but a monetary value does not. Parse money with json.loads(text, parse_float=decimal.Decimal) when exactness matters, and keep large IDs as strings if you will ever hand them back to a JavaScript client.

Duplicate keys silently lose data — {"a": 1, "a": 2} is legal JSON, and json.loads keeps the last value. If a vendor feed does this deliberately to carry repeated fields, pass object_pairs_hook=list and handle the pairs yourself; otherwise treat it as a corruption signal worth logging.

lxml.etree.XMLSyntaxError: Detected an entity reference loop — the document contains a billion-laughs style entity expansion. lxml blocks this by default; if you disabled resolve_entities=False to work around a vendor feed, re-enable it rather than raising the expansion limit.

Frequently Asked Questions

When is xmltodict better than lxml?xmltodict is best for small-to-medium XML you want as plain dicts with no XPath — feeds, sitemaps under a few megabytes, simple vendor APIs. Switch to lxml as soon as the document declares namespaces you need to query, exceeds roughly 50 MB, or requires real XPath predicates, because xmltodict builds the whole structure in Python objects and cannot stream.

How do I handle XML namespaces in XPath? Declare a prefix-to-URI mapping and use that prefix in the query, for example root.findall(".//sm:loc", {"sm": "http://www.sitemaps.org/schemas/sitemap/0.9"}). Without the mapping the query matches nothing even though the elements are visibly present, because libxml2 stores the fully qualified name internally. Inspect root.nsmap when you are unsure which URIs a document declares.

Why does my JSON parse fail even though the request succeeded? A 200 status does not guarantee a JSON body — anti-bot pages, redirects to a login form, and CDN error pages routinely return HTML with a success code. Verify the Content-Type header and peek at the first 200 characters of response.text before parsing, and treat a mismatch as a retry-worthy failure rather than a parse error.

What is JSONPath and do I actually need it? JSONPath is a query language for JSON, roughly analogous to XPath for XML. You do not strictly need it, but for deeply nested or irregular payloads a single expression such as $.results[*].variants[*].price is clearer and safer than nested loops, and it returns an empty list rather than raising when part of the path is missing.

Can I parse a 2 GB XML file without a 2 GB machine? Yes, with lxml.etree.iterparse plus explicit cleanup. Handle each record on the end event, call elem.clear(), and delete the already-processed preceding siblings so the parent does not retain empty shells. Done correctly, resident memory stays roughly constant no matter how long the document is.

Related

• Data Extraction Patterns and APIs — the parent section covering every extraction route
• Parsing XML Sitemaps with Python — index files, gzip, and the 50,000-URL limit
• Flattening Nested JSON with pandas — turning nested payloads into tabular rows
• Extracting JSON-LD and Structured Data — JSON embedded in ordinary HTML pages
• Scraping GraphQL Endpoints — when the JSON comes back from a single POST