Scraping schema.org Product Data from JSON-LD

This walkthrough zooms in on the highest-value target from Extracting JSON-LD and Structured Data: pulling price, availability and ratings out of Product nodes across e-commerce pages.

A schema.org Product nests price and availability under offers, and ratings under aggregateRating.

Most online stores publish a schema.org Product block so their listings qualify for rich search results, and that block is the single best place to read price and stock from. Price and availability live nested inside an offers object, ratings live inside aggregateRating, and numbers are almost always stored as strings. Handle those three shapes correctly and you get a clean product record from nearly any retailer with one request and no site-specific selectors.

How the Product Node Is Structured

A typical Product node looks like this once parsed into a Python dict:

{
  "@type": "Product",
  "name": "Aeron Chair",
  "sku": "AER-001",
  "gtin13": "0885915430215",
  "brand": {"@type": "Brand", "name": "Herman Miller"},
  "offers": {
    "@type": "Offer",
    "price": "1395.00",
    "priceCurrency": "USD",
    "priceValidUntil": "2026-12-31",
    "availability": "https://schema.org/InStock",
    "itemCondition": "https://schema.org/NewCondition"
  },
  "aggregateRating": {
    "@type": "AggregateRating",
    "ratingValue": "4.7",
    "reviewCount": "212",
    "bestRating": "5"
  }
}

Four things trip people up. Price is nested: it is node["offers"]["price"], never node["price"]. offers is polymorphic — a single object, a list of objects, or an AggregateOffer that has lowPrice and highPrice and no price key at all. Availability is a URL, not a boolean, and the enumeration has seven members, not two. And bestRating is not always 5: a site rating out of 10 will hand you "ratingValue": "8.4", which silently corrupts any comparison you do across retailers.

offers is typed as Offer or AggregateOffer, and either may appear as a list, so a parser that assumes a single object with a price key breaks on two of the three real shapes.

The availability enumeration is worth knowing in full, because treating anything that is not InStock as out of stock loses real signal: InStock, OutOfStock, PreOrder, BackOrder, LimitedAvailability, SoldOut, and Discontinued. Some sites also emit the older http://schema.org/InStock with http rather than https, and a handful emit the bare token with no URL prefix at all, which is why the parser below reduces every form to its last path segment.

A Robust Product Parser

The function below fetches a page, isolates the Product node, and flattens the nested offers and aggregateRating into a single typed record. It casts stringified numbers, normalises the availability URL, and rescales the rating when bestRating is not 5.

import json
import re
import requests
from bs4 import BeautifulSoup

HEADERS = {
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 "
                  "(KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36",
    "Accept": "text/html,application/xhtml+xml",
}
AVAILABILITY = {
    "InStock", "OutOfStock", "PreOrder", "BackOrder",
    "LimitedAvailability", "SoldOut", "Discontinued",
}
NUMERIC_RE = re.compile(r"-?\d[\d.,\s]*")


def to_float(value: object) -> float | None:
    """Cast a schema.org number, tolerating currency symbols and locale separators."""
    if value is None:
        return None
    if isinstance(value, (int, float)):
        return float(value)
    match = NUMERIC_RE.search(str(value))
    if not match:
        return None
    text = match.group(0).strip().replace(" ", "")
    if "," in text and "." in text:                     # 1,395.00 or 1.395,00
        decimal = "," if text.rfind(",") > text.rfind(".") else "."
        text = text.replace("," if decimal == "." else ".", "").replace(decimal, ".")
    elif text.count(",") == 1 and len(text.split(",")[-1]) in (1, 2):
        text = text.replace(",", ".")                   # 1395,00 -> 1395.00
    else:
        text = text.replace(",", "")                    # 1,395 -> 1395
    try:
        return float(text)
    except ValueError:
        return None


def pick_offer(offers: object) -> dict:
    """offers may be one Offer, a list of them, or an AggregateOffer."""
    if isinstance(offers, list):
        priced = [o for o in offers if isinstance(o, dict) and to_float(o.get("price")) is not None]
        if priced:
            return min(priced, key=lambda o: to_float(o["price"]) or float("inf"))
        return offers[0] if offers and isinstance(offers[0], dict) else {}
    return offers if isinstance(offers, dict) else {}


def parse_product(node: dict) -> dict[str, object]:
    offer = pick_offer(node.get("offers"))
    rating = node.get("aggregateRating") or {}
    token = str(offer.get("availability", "")).rsplit("/", 1)[-1].lstrip("#")
    price = to_float(offer.get("price"))
    if price is None:
        price = to_float(offer.get("lowPrice"))          # AggregateOffer
    brand = node.get("brand")
    best = to_float(rating.get("bestRating")) or 5.0
    value = to_float(rating.get("ratingValue"))
    return {
        "name": node.get("name"),
        "sku": node.get("sku") or node.get("gtin13"),
        "brand": brand.get("name") if isinstance(brand, dict) else brand,
        "price": price,
        "currency": offer.get("priceCurrency"),
        "availability": token if token in AVAILABILITY else None,
        "in_stock": token in {"InStock", "LimitedAvailability", "PreOrder"},
        "rating_out_of_5": round(value / best * 5, 2) if value is not None else None,
        "review_count": int(to_float(rating.get("reviewCount")) or 0) or None,
    }


def product_from_url(url: str) -> dict[str, object] | None:
    resp = requests.get(url, headers=HEADERS, timeout=15)
    resp.raise_for_status()
    soup = BeautifulSoup(resp.text, "lxml")
    for block in soup.find_all("script", type="application/ld+json"):
        raw = block.string or block.get_text()
        if not raw:
            continue
        try:
            parsed = json.loads(raw)
        except json.JSONDecodeError:
            continue
        candidates = parsed if isinstance(parsed, list) else [parsed]
        for entry in candidates:
            if not isinstance(entry, dict):
                continue
            for node in entry.get("@graph", [entry]):
                declared = node.get("@type", "")
                types = declared if isinstance(declared, list) else [declared]
                if any(str(t).rsplit("/", 1)[-1] == "Product" for t in types):
                    return parse_product(node)
    return None


if __name__ == "__main__":
    print(product_from_url("https://example.com/product/aeron-chair"))

The output is a flat dict with a float price, a bool stock flag and a rating already rescaled to a five-point scale, so records from different retailers are directly comparable. Two design choices are deliberate. Choosing the lowest priced entry when offers is a list matches what a shopper sees on a marketplace listing; if you are tracking a specific seller, filter on offer["seller"]["name"] before the min. And treating PreOrder as in stock is a judgement call — it is orderable but not shippable — so decide once and record the raw token alongside the boolean.

Handling the Fields That Are Simply Missing

Even correct parsers return None for a fraction of pages. What matters is knowing which fallback produced each value, because a shift in the mix is the earliest signal that a site changed its markup.

Walk the chain in order and record which rung produced the value. If the mix of rungs shifts between runs, the site changed its markup before your extractor noticed.

def price_with_source(node: dict) -> tuple[float | None, str]:
    offer = pick_offer(node.get("offers"))
    if isinstance(node.get("offers"), dict) and to_float(offer.get("price")) is not None:
        return to_float(offer["price"]), "offers.price"
    if isinstance(node.get("offers"), list) and to_float(offer.get("price")) is not None:
        return to_float(offer["price"]), "offers[].price"
    if to_float(offer.get("lowPrice")) is not None:
        return to_float(offer["lowPrice"]), "offers.lowPrice"
    return None, "missing"

Log the source string with every record and aggregate it per domain. When offers.price drops from 78% to 3% of a retailer's pages overnight, that is a markup change, not a stock event — the kind of silent failure covered in Detecting Silent Scraper Failures.

Scaling Across Many Product Pages

Because every retailer using schema.org shares the same field names, one parser works across sites with only small adjustments. Loop a list of product URLs, collect the dicts, and hand the batch to storage via Storing and Exporting Scraped Data. If you gather listings from a JSON API rather than crawling category pages, pair this with Parsing JSON and XML Responses to get the URLs first, then enrich each with its JSON-LD. Before the records reach a database, run them through the type and range checks in Cleaning and Validating Scraped Data — a price of 0.0 or 999999 is far more often a parsing artefact than a real listing.

Cross-Checking Against the Rendered Page

Structured data is generated by a template, and templates go stale. The two failure modes worth catching are a JSON-LD price that no longer matches the price the page displays, and a block cached from a previous deploy. Neither raises an exception, so the only way to find them is to compare.

import re
from bs4 import BeautifulSoup

PRICE_TEXT_RE = re.compile(r"[£$€]\s?\d[\d.,]*")


def visible_prices(html: str) -> list[float]:
    soup = BeautifulSoup(html, "lxml")
    for tag in soup(["script", "style", "noscript"]):
        tag.decompose()
    found = PRICE_TEXT_RE.findall(soup.get_text(" ", strip=True))
    return [p for p in (to_float(t) for t in found) if p is not None]


def price_agrees(structured: float | None, html: str, tolerance: float = 0.01) -> bool:
    if structured is None:
        return False
    return any(abs(structured - shown) <= tolerance for shown in visible_prices(html))

Run this on a sample — 50 pages per domain is plenty — rather than on every request; it costs a full text extraction per page. What you are watching is the agreement rate, not any individual result. A domain that agrees on 95% of sampled pages is healthy; one that drops to 40% has changed something, and the structured block is now describing a different product state than the page. Stripping script and style before the text search matters, or the regular expression will match the very JSON-LD you are trying to verify against and always agree with itself.

Edge Cases and Caveats

Multiple offers. Marketplaces list several sellers under offers. Decide whether you want the lowest price, the featured one, or a specific merchant, and record the count so a single-seller page and a fifty-seller page are distinguishable.
AggregateOffer instead of Offer. Some pages set @type: AggregateOffer with lowPrice, highPrice and offerCount and no price key. Check for those before concluding the price is missing.
Missing aggregateRating. New or unreviewed products omit the block entirely, so a lookup returns nothing. Use .get(...) with defaults; None reviews and zero reviews are different facts and should stay distinguishable.
Locale number formats. Prices arrive as "1,395.00", "1.395,00", "1 395,00 €" and occasionally "US$1,395". The to_float helper above disambiguates by which separator appears last, which handles every common European and US format.
Currency mismatches. priceCurrency can differ from the currency the page displays when a site geolocates, and it is sometimes absent entirely. Always store the code alongside the number rather than assuming one, and treat a missing code as unknown rather than defaulting to USD.
Stale priceValidUntil. A date in the past usually means the block is cached or generated from a stale feed. It is a useful freshness signal even though nothing enforces it.
JavaScript-injected blocks. If the Product JSON-LD is absent from the raw HTML, it is rendered client-side. Render the page first or read the product API directly using the workflow in Reverse-Engineering Private APIs.
sku is not a stable identifier across sites. Two retailers use different SKUs for the same physical item, and some use the internal database id. gtin13, gtin14 and mpn are the fields that let you match across retailers, so read them when present and store all of them rather than collapsing to one.
priceCurrency and the price can disagree with offers.priceSpecification. When both are present, priceSpecification usually carries the more precise figure, including whether tax is included via valueAddedTaxIncluded. A price that looks 20% off between two sources is often exactly this.
Variant pages. A product with sizes or colours may publish one Product per variant, or one ProductGroup with hasVariant. Reading only the first node then silently records one variant's price as the product's price.

Frequently Asked Questions

Where exactly is the price in a schema.org Product? Inside the offers object, as product["offers"]["price"], and almost always as a string that needs casting. Remember that offers may be a list of offers from different sellers, or an AggregateOffer that exposes lowPrice and highPrice instead of price.

How do I tell if a product is in stock from JSON-LD? Read offers.availability, which is a schema.org URL such as https://schema.org/InStock. Take the segment after the last slash and compare it against the full enumeration, which includes PreOrder, BackOrder, LimitedAvailability and SoldOut as well as the two obvious values.

Why are the ratings missing on some pages? Products with no reviews omit aggregateRating entirely, so the lookup returns nothing. Use .get("aggregateRating", {}) and keep None distinct from zero — a product with no reviews and a product rated zero are different states, and merging them distorts any average you compute later.

Can I use this same parser on different e-commerce sites? Largely yes, because schema.org field names are standardised — name, offers.price and aggregateRating.ratingValue mean the same thing everywhere. What varies is whether offers is a list, whether the type is AggregateOffer, what bestRating is, and how numbers are formatted for the locale, all of which the parser above already handles.

Extracting JSON-LD and Structured Data — the parent topic and the other embedded syntaxes
Flattening Nested JSON with pandas — turning batches of these records into a table
Normalizing Prices, Dates and Units — currency conversion and locale-safe casting
Scraping GraphQL Endpoints — when the store exposes an API instead