Extracting JSON-LD and Structured Data with Python

Search engines do not guess what a page is about — they read structured data the site publishes on purpose, and that same data is the cleanest extraction target available. This guide belongs to Data Extraction Patterns and APIs and covers the four embedded syntaxes you will meet in the wild: JSON-LD, microdata, RDFa and Open Graph. Instead of chasing CSS classes through presentation markup, you read a documented application/ld+json block and get a typed object back, using BeautifulSoup, the standard-library json module, and the extruct library that unifies all four.

When to Use Structured Data Extraction

Reach for embedded structured data before writing a single visible-markup selector when any of the following holds:

• The page is an e-commerce product, recipe, article, event, job posting, or local business listing. These page types compete for rich search results, so structured data is close to universal on them.
• You need extraction that survives redesigns. A front-end rebuild rewrites class names and DOM nesting; it rarely rewrites the JSON-LD block, because that block is generated from the same backend model that feeds the API.
• You want fields that are awkward to select visually — SKU, GTIN, ISO 8601 dates, currency codes, @id references, geo coordinates — that are explicit in the structured block but implied or absent in the rendered text.
• You are crawling many unfamiliar domains and cannot write a bespoke selector set for each one. Structured data gives you a single extractor that works across thousands of sites.
• You already have records from an API and want a second, independent source of truth to cross-check prices or titles.

Fall back to visible-HTML parsing only when a page publishes nothing at all, or when the block omits a field you need. That fallback path — attribute reading, nested tag traversal, and the selector patterns that hold up over time — is covered in Extracting Attributes and Nested Tags.

The ranking above is not arbitrary. JSON-LD is a self-contained document, so a change to the page layout cannot break it. Microdata and RDFa annotate the visible elements, so a template refactor that moves an itemprop onto a different tag changes what a naive parser reads. Open Graph is designed for social previews and rarely carries more than a title, description and image. Visible-markup selectors are last because they encode assumptions about a specific site's DOM.

Prerequisites

Use Python 3.10 or newer. Install a parser, the HTTP client, and extruct:

python -m pip install "requests>=2.31" "beautifulsoup4>=4.12" "lxml>=5.0" "extruct>=0.17"

requests fetches pages, beautifulsoup4 with the lxml tree builder locates the blocks, and extruct extracts every structured-data syntax in one call when you want them all at once. extruct pulls in pyrdfa3, mf2py and html-text as dependencies, which is why it is a heavier import — about 80 ms of startup cost — than a bare BeautifulSoup workflow. If you only ever need JSON-LD, skip it.

Step-by-Step: Reading Every Syntax a Page Publishes

1. Find and Parse JSON-LD Blocks

JSON-LD lives in <script type="application/ld+json"> tags. A single page routinely has three or four of them, and each may be a single object, an array of objects, or an object wrapping a @graph list. Parse defensively and flatten everything to one list before you filter.

import json
import requests
from bs4 import BeautifulSoup

HEADERS = {
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 "
                  "(KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36",
    "Accept": "text/html,application/xhtml+xml",
}


def extract_jsonld(html: str) -> list[dict]:
    soup = BeautifulSoup(html, "lxml")
    nodes: list[dict] = []
    for block in soup.find_all("script", type="application/ld+json"):
        raw = block.string or block.get_text()
        if not raw or not raw.strip():
            continue
        try:
            parsed = json.loads(raw)
        except json.JSONDecodeError:
            continue                        # skip one broken block, keep the page
        nodes.extend(_flatten(parsed))
    return nodes


def _flatten(parsed: object) -> list[dict]:
    if isinstance(parsed, list):
        out: list[dict] = []
        for entry in parsed:
            out.extend(_flatten(entry))
        return out
    if isinstance(parsed, dict):
        if "@graph" in parsed:
            return _flatten(parsed["@graph"])
        return [parsed]
    return []


def fetch_jsonld(url: str) -> list[dict]:
    resp = requests.get(url, headers=HEADERS, timeout=15)
    resp.raise_for_status()
    return extract_jsonld(resp.text)

The recursion in _flatten matters because the three container shapes nest: a list of objects where one of them holds a @graph is legal and appears on WordPress sites with several SEO plugins active. Splitting the fetch from the parse also lets you unit-test the extractor against saved HTML fixtures without a network call, which is the difference between a five-second test suite and a five-minute one.

Note block.string or block.get_text(). Tag.string returns None whenever the script element contains more than one child node, which happens when a CDN injects a comment into the block. get_text() concatenates all descendants and gives you the payload anyway.

2. Resolve @id References Inside a @graph

Sites that emit a @graph usually normalise repeated entities: the Product node does not embed the brand, it points at it. Reading node["brand"]["name"] then returns KeyError: 'name', because brand is {"@id": "https://example.com/#brand"} and nothing else.

def index_by_id(nodes: list[dict]) -> dict[str, dict]:
    return {node["@id"]: node for node in nodes if isinstance(node.get("@id"), str)}


def resolve(value: object, index: dict[str, dict]) -> object:
    """Replace a bare {"@id": ...} reference with the node it points at."""
    if isinstance(value, list):
        return [resolve(item, index) for item in value]
    if isinstance(value, dict) and set(value) == {"@id"}:
        return index.get(value["@id"], value)
    return value


def brand_name(product: dict, index: dict[str, dict]) -> str | None:
    brand = resolve(product.get("brand"), index)
    if isinstance(brand, dict):
        return brand.get("name")
    if isinstance(brand, str):
        return brand                        # schema.org allows a plain string here
    return None

The set(value) == {"@id"} test is deliberately strict: a node that carries both @id and real fields is already inlined and must not be replaced. Note also that schema.org types brand as Brand or Organization, and many sites simply write "brand": "Herman Miller". Any field on a schema.org type can legally be a string, an object, or a list of either, which is the single biggest source of TypeError in structured-data parsers.

3. Filter by schema.org Type

Structured blocks are self-describing through their @type. Once you have the flat list, pick the nodes you care about — Product, Article, Recipe, Event, JobPosting.

def nodes_of_type(nodes: list[dict], schema_type: str) -> list[dict]:
    matches: list[dict] = []
    for node in nodes:
        declared = node.get("@type", "")
        types = declared if isinstance(declared, list) else [declared]
        # @type may be a bare name or a full IRI: "Product" or "http://schema.org/Product"
        short = {str(t).rsplit("/", 1)[-1].lstrip("#") for t in types}
        if schema_type in short:
            matches.append(node)
    return matches


if __name__ == "__main__":
    data = fetch_jsonld("https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html")
    print(nodes_of_type(data, "Product"))

Two normalisations are doing work here. @type is frequently a list — a node can be both Product and IndividualProduct — and it is sometimes written as a full IRI rather than the short name, particularly on RDFa-derived output. Comparing on the last path segment handles both without a dependency on a JSON-LD processor.

Pulling specific fields such as price, availability and ratings out of Product nodes has enough nesting of its own to deserve separate treatment — see Scraping schema.org Product Data for the full field map, the offers list case, and AggregateOffer.

4. Read Microdata and Open Graph Tags

Not every site uses JSON-LD. Older or CMS-driven pages often use microdata (itemscope, itemtype and itemprop attributes) or, for social previews, Open Graph meta tags. Open Graph is trivial to read directly:

def open_graph(html: str) -> dict[str, str]:
    soup = BeautifulSoup(html, "lxml")
    og: dict[str, str] = {}
    for meta in soup.find_all("meta"):
        key = meta.get("property") or meta.get("name") or ""
        content = meta.get("content")
        if key.startswith("og:") and content:
            og[key[3:]] = content           # "og:title" -> "title"
    return og

The property or name fallback matters: the Open Graph specification says property, but a large minority of sites — and every page generated by certain older CMS templates — emit name="og:title" instead. Reading only property silently returns an empty dict on those pages.

Microdata is more work by hand, because a value can come from content, href, src, datetime or the element's text depending on the tag:

from bs4 import Tag

VALUE_ATTRS = {
    "meta": "content", "a": "href", "link": "href", "img": "src",
    "audio": "src", "video": "src", "time": "datetime", "data": "value",
}


def microdata_value(el: Tag) -> str | None:
    attr = VALUE_ATTRS.get(el.name)
    if attr and el.get(attr):
        return str(el[attr]).strip()
    return el.get_text(strip=True) or None


def microdata_item(scope: Tag) -> dict[str, str | None]:
    item: dict[str, str | None] = {"@type": scope.get("itemtype")}
    for prop in scope.find_all(attrs={"itemprop": True}):
        # skip properties that belong to a nested itemscope
        if prop.find_parent(attrs={"itemscope": True}) is not scope:
            continue
        item[str(prop["itemprop"])] = microdata_value(prop)
    return item

The nested-scope guard is the part most hand-rolled microdata readers omit, and it is why a product's name sometimes comes back as the reviewer's name: find_all descends into child itemscope elements, so an inner Review item's properties get merged into the outer Product.

5. Extract Every Format at Once with extruct

When you do not know in advance which syntax a site uses, extruct reads all of them in a single pass and returns a dictionary keyed by format. It is the pragmatic choice for crawling many unfamiliar domains.

import extruct
import requests
from w3lib.html import get_base_url

HEADERS = {
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 "
                  "(KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36",
    "Accept": "text/html,application/xhtml+xml",
}
SYNTAXES = ["json-ld", "microdata", "rdfa", "opengraph"]


def extract_all(url: str) -> dict[str, list]:
    resp = requests.get(url, headers=HEADERS, timeout=15)
    resp.raise_for_status()
    base_url = get_base_url(resp.text, str(resp.url))
    return extruct.extract(resp.text, base_url=base_url, syntaxes=SYNTAXES, uniform=True)


def best_product(url: str) -> dict | None:
    data = extract_all(url)
    for syntax in SYNTAXES:
        for node in data.get(syntax, []):
            declared = str(node.get("@type", ""))
            if declared.rsplit("/", 1)[-1] == "Product":
                return node
    return None

uniform=True is worth setting on every call. Without it, microdata items come back with type and properties keys while JSON-LD comes back with @type and flat fields, so you end up writing two extractors anyway. With it, extruct rewrites microdata and RDFa into the JSON-LD shape and one downstream function handles all of them. Passing an accurate base_url is what lets RDFa and microdata resolve relative href and src values into absolute URLs.

6. Read the Types Beyond Product

Product gets the attention, but the same extractor covers several other high-value types with only a different field map. Knowing the field names saves you from inspecting each site by hand.

Two of these repay a second look. ISO 8601 durations such as PT1H30M are not parseable by datetime; use isodate.parse_duration or a small regular expression, because float("PT1H30M") obviously fails and int(...[2:-1]) quietly returns the wrong number for anything over an hour.

BreadcrumbList is underused. It gives you the site's own category hierarchy as an ordered list of URLs, which is a far more reliable crawl map than guessing from the navigation menu:

def breadcrumb_trail(nodes: list[dict]) -> list[tuple[int, str, str | None]]:
    trail: list[tuple[int, str, str | None]] = []
    for node in nodes_of_type(nodes, "BreadcrumbList"):
        for element in node.get("itemListElement") or []:
            item = element.get("item")
            url = item.get("@id") if isinstance(item, dict) else item
            name = item.get("name") if isinstance(item, dict) else element.get("name")
            trail.append((int(element.get("position", 0)), str(name or ""), url))
    return sorted(trail)

Sorting on position matters because the elements are not required to appear in order in the array, and several popular SEO plugins emit them out of sequence.

7. Validate Before You Store

Structured data is published by marketing tooling, so it is well-formed far more often than it is correct. Prices appear as "1,395.00", "USD 1395" and 1395; availability is a URL on some sites and the bare token InStock on others; dates arrive with and without a timezone offset. Coerce at the boundary and reject rather than guess:

def normalise_availability(raw: object) -> str | None:
    if not raw:
        return None
    token = str(raw).rsplit("/", 1)[-1].lstrip("#")
    known = {"InStock", "OutOfStock", "PreOrder", "BackOrder",
             "Discontinued", "LimitedAvailability", "SoldOut"}
    return token if token in known else None

Anything that fails this check should be recorded as unknown rather than defaulted to in-stock, because a silent default turns a parsing bug into a business decision. The wider toolkit — declarative models, unit and currency normalisation, and de-duplication across sources — is in Cleaning and Validating Scraped Data.

Performance and Scaling Considerations

Structured-data extraction is cheap: you are already downloading the HTML, and parsing one small <script> block costs far less than walking the full DOM with dozens of selectors. A few things keep it fast at volume.

• Use the lxml tree builder. On a 300 KB product page, BeautifulSoup(html, "lxml") is roughly 3–5× faster than html.parser; the measured trade-offs are laid out in BeautifulSoup vs lxml: Which Parser Is Faster.
• Skip the whole tree when you can. For JSON-LD only, a compiled regular expression that slices out <script type="application/ld+json">…</script> avoids building a DOM at all and typically runs 10–20× faster per page. It is less robust against attribute-order variation, so keep the BeautifulSoup path as the fallback when the regex finds nothing.
• Reserve extruct for unknown sites. Running four extractors costs roughly 4–8× a targeted JSON-LD parse. Once you know a domain publishes JSON-LD, pin the extractor to that syntax.
• Do not retain soup objects. A BeautifulSoup tree for a large page holds several megabytes. Extract the dicts you need and let the tree go out of scope before the next fetch, or a concurrent crawler will grow its resident set with every in-flight page.
• Batch the writes. The output is typed dicts, so it flows straight into Storing and Exporting Scraped Data with almost no reshaping.
• Fetch with a session, not bare calls. Structured-data crawls hit many pages on the same host, so connection reuse dominates. A requests.Session or an httpx.Client removes a TLS handshake per page, which on a thousand-page domain is minutes rather than milliseconds.
• Do not re-parse for each field. Build the flat node list once per page and pass it to every field extractor. Calling a helper that re-runs BeautifulSoup internally is the most common accidental quadratic in this kind of code.

Common Errors and Fixes

json.JSONDecodeError: Expecting value — the block contains trailing commas, HTML comments, or CDATA wrappers that are not valid JSON. Strip the wrappers and keep going rather than aborting the page:

raw = (block.string or block.get_text() or "").strip()
if raw.startswith("<!--"):
    raw = raw.removeprefix("<!--").removesuffix("-->").strip()
raw = raw.removeprefix("//<![CDATA[").removesuffix("//]]>").strip()
try:
    parsed = json.loads(raw)
except json.JSONDecodeError:
    parsed = None

TypeError: 'NoneType' object is not subscriptable — block.string is None because the script tag holds nested nodes or is empty. Use block.string or block.get_text() as shown throughout this guide.

TypeError: string indices must be integers on node["brand"]["name"] — brand is a plain string on this site. Every schema.org property can be a string, an object, or a list, so type-test before subscripting.

KeyError: '@type' — a node has no @type, or it is a list rather than a string. Always read it with .get("@type", "") and normalise a list-or-string into a set before comparing.

KeyError: 'name' on a resolved reference — the value was {"@id": "..."} and the real node lives elsewhere in the @graph. Build the @id index and resolve first.

Empty results on a page that visibly has rich snippets — the JSON-LD is injected by JavaScript after load, so it is absent from the initial HTML. Confirm by searching the raw response for ld+json; if it is not there, either find the API that supplies it or render the page first with Playwright and run the same extractor on the rendered DOM.

The same product appears twice with different values — the page carries both a JSON-LD Product and a microdata Product, and they disagree because one is generated from the catalogue and the other from the template. Prefer JSON-LD, but record which syntax supplied each field so the disagreement is visible rather than arbitrary.

@context is not schema.org — a node whose context points at a different vocabulary uses the same key names for different meanings. Read parsed.get("@context") and skip anything that is not schema.org (with or without the https and the trailing slash) before you map fields.

extruct raises AttributeError inside pyrdfa3 on a malformed page — the RDFa extractor is the least tolerant of the four. Drop "rdfa" from syntaxes for that domain, or call extruct.extract per syntax inside a try/except so one failing extractor does not lose the other three.

Frequently Asked Questions

What is the difference between JSON-LD, microdata, and RDFa? All three encode schema.org vocabulary, but JSON-LD keeps the data in a separate <script> block that is independent of the page layout, while microdata and RDFa annotate the visible HTML with attributes. JSON-LD is the format Google recommends and the one you will encounter most, and it is the only one of the three that cannot be broken by a template refactor.

Do I need extruct, or is BeautifulSoup enough? For JSON-LD alone, BeautifulSoup plus json.loads is enough, lighter, and easier to debug. Use extruct with uniform=True when a page might use microdata or RDFa, or when you are crawling many sites and do not want to maintain a separate extractor for each syntax.

Why is the JSON-LD block missing from my downloaded HTML? Some sites inject structured data with JavaScript after the initial load, so it never appears in the response requests receives. Search the raw response text for ld+json to confirm; if it is absent, the data is added client-side and you will need to render the page or locate the API that feeds it.

How do I handle a field that is sometimes a string and sometimes an object? Type-test it. Schema.org properties are typed as unions, so brand, author, offers and image all legally accept a string, an object, or a list of either. A small helper that normalises any of those into a list of dicts, and treats a bare string as {"name": value}, removes an entire class of TypeError.

Is structured data ever deliberately wrong? Occasionally. Sites sometimes publish a list price in JSON-LD while showing a discounted price on the page, or leave availability stale after a stock update. When the number matters, cross-check the structured value against the rendered text on a sample of pages and alert if they diverge on more than a small fraction.

Related

• Data Extraction Patterns and APIs — the parent section and the other extraction routes
• Scraping schema.org Product Data — price, stock and rating fields in detail
• Parsing JSON and XML Responses — handling the JSON once you have it
• Reverse-Engineering Private APIs — where to go when the block is rendered client-side
• Cleaning and Validating Scraped Data — coercing and rejecting the values you extracted