Scrapy vs Playwright for Single-Page Apps

Single-page apps defeat naive HTML scrapers because the content arrives after JavaScript runs, and this page — part of Web Scraping with Scrapy — explains when plain Scrapy still wins that fight and when a browser is genuinely unavoidable.

For an SPA, prefer Scrapy calling the hidden JSON API; fall back to Playwright rendering only when client-side execution is unavoidable.

For most single-page apps you do not need a browser at all. Whatever renders on screen was fetched by the page's own JavaScript from a JSON endpoint, and you can call that endpoint directly from Scrapy — faster, cheaper, more stable, and returning structured data instead of HTML to parse. Reserve Playwright for cases where the data depends on client-side execution you cannot replay: request signing by obfuscated JavaScript, canvas or WebGL output, or state assembled across interdependent requests. The middle path is scrapy-playwright, which renders only the requests that need it inside an otherwise normal crawl.

Why the Shell Is Empty

A single-page app ships a near-empty HTML document and a JavaScript bundle. The browser executes the bundle, which issues fetch or XMLHttpRequest calls, receives JSON, and writes elements into the DOM. Scrapy downloads the document and stops there, so response.css("div.product") matches nothing — not because the selector is wrong, but because the element does not exist yet.

A browser walks four steps to put data on the screen. A spider that calls the same endpoint directly starts at step three and skips the shell, the bundle and the render entirely.

That diagram contains the whole strategy. The browser walks four steps to get data on screen; three of them exist to make a human-usable page and produce no data you need. A spider that issues the step-three request directly gets the same payload with none of the surrounding cost, and gets it as JSON rather than as markup it has to reverse-engineer.

The counter-argument — "but the page might change its markup" — actually favours the API path. Private JSON endpoints are consumed by the site's own front end, so they change on the front end's release schedule and their field names are stable between releases. Markup, by contrast, changes whenever a designer touches a component. Finding the endpoint is covered step by step in Finding Hidden API Endpoints in Network Traffic.

Option A: Call the API from Scrapy

Open developer tools, switch to the Network tab, filter to Fetch/XHR and reload. Look for a response whose JSON matches what is on screen. Copy it as cURL to capture the exact headers, then reproduce it in Scrapy — usually you can drop most headers and keep Accept, User-Agent, and whatever token or Referer the endpoint checks.

Pagination, filtering and sorting normally become query-string parameters at this point, which makes them far easier than the DOM equivalents described in Handling Pagination and Infinite Scroll.

# api_spider.py — run with: scrapy runspider api_spider.py -o items.jsonl
import json
from collections.abc import Iterator

import scrapy


class ProductApiSpider(scrapy.Spider):
    name = "product_api"
    custom_settings = {
        "DEFAULT_REQUEST_HEADERS": {
            "User-Agent": (
                "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 "
                "(KHTML, like Gecko) Chrome/125.0 Safari/537.36"
            ),
            "Accept": "application/json",
            "Referer": "https://example.com/products",
        },
        "CONCURRENT_REQUESTS_PER_DOMAIN": 8,
        "DOWNLOAD_DELAY": 0.2,
    }

    def start_requests(self) -> Iterator[scrapy.Request]:
        yield scrapy.Request(
            "https://example.com/api/products?page=1&page_size=50",
            callback=self.parse,
            cb_kwargs={"page": 1},
        )

    def parse(self, response: scrapy.http.Response, page: int) -> Iterator[dict | scrapy.Request]:
        try:
            payload = response.json()
        except json.JSONDecodeError:
            self.logger.error("non-JSON response at page %s: %s", page, response.text[:200])
            return

        results = payload.get("results", [])
        for item in results:
            yield {
                "id": item["id"],
                "title": item["name"],
                "price": item["price"],
            }

        if results and payload.get("next"):
            yield response.follow(payload["next"], callback=self.parse, cb_kwargs={"page": page + 1})

Two things in that spider exist because private APIs misbehave. The JSONDecodeError guard catches the case where the endpoint answers a rate-limit or challenge page in HTML — otherwise the spider dies with a confusing traceback several hundred pages in. And following payload["next"] rather than incrementing a counter means the crawl stops when the server says it is done, instead of paging into empty results forever.

Items from here flow into normal Scrapy machinery: validation, deduplication and storage in an item pipeline, as in Writing Scrapy Item Pipelines, then into a durable sink per Storing and Exporting Scraped Data.

Option B: The scrapy-playwright Hybrid

Sometimes the endpoint is signed by client JavaScript, or the value you need is computed in the browser. Rather than rewriting the project around a browser, add scrapy-playwright and mark only the requests that need rendering. Everything else stays on the fast HTTP path, and you keep Scrapy's scheduler, retries, throttling and pipelines for the whole crawl.

# hybrid_spider.py — pip install scrapy-playwright && playwright install chromium
from collections.abc import Iterator

import scrapy


class SpaHybridSpider(scrapy.Spider):
    name = "spa_hybrid"
    custom_settings = {
        "TWISTED_REACTOR": "twisted.internet.asyncioreactor.AsyncioSelectorReactor",
        "DOWNLOAD_HANDLERS": {
            "http": "scrapy_playwright.handler.ScrapyPlaywrightDownloadHandler",
            "https": "scrapy_playwright.handler.ScrapyPlaywrightDownloadHandler",
        },
        "PLAYWRIGHT_LAUNCH_OPTIONS": {"headless": True},
        "PLAYWRIGHT_MAX_CONTEXTS": 4,
        "PLAYWRIGHT_DEFAULT_NAVIGATION_TIMEOUT": 20000,
        "CONCURRENT_REQUESTS": 16,
        "DEFAULT_REQUEST_HEADERS": {
            "User-Agent": (
                "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 "
                "(KHTML, like Gecko) Version/17.4 Safari/605.1.15"
            ),
        },
    }

    def start_requests(self) -> Iterator[scrapy.Request]:
        yield scrapy.Request(
            "https://example.com/dashboard",
            meta={
                "playwright": True,
                "playwright_include_page": False,
                "playwright_page_methods": [
                    {"method": "wait_for_selector", "args": ["div.report-row"]},
                ],
            },
            callback=self.parse,
        )

    def parse(self, response: scrapy.http.Response) -> Iterator[dict]:
        for row in response.css("div.report-row"):
            yield {
                "label": row.css("::attr(data-label)").get(),
                "value": row.css("span.value::text").get(),
            }

Only requests carrying meta={"playwright": True} launch a browser. PLAYWRIGHT_MAX_CONTEXTS is the setting that stops the crawl from opening a browser context per concurrent request and exhausting memory — set it well below CONCURRENT_REQUESTS, because the HTTP requests can safely run far wider than the rendered ones.

If rendering triggers a challenge rather than the page you expected, the relevant work is in Bypassing Cloudflare and Akamai Protections — a default Playwright build is detectable, and rendering by itself does not make a request look human.

What the Browser Actually Costs

The hybrid is affordable because cost is paid per rendered request, not per crawl.

In a hybrid crawl the browser cost is paid per rendered request, not per crawl. Keeping the rendered share small is what keeps the memory and time budget within reach of one machine.

A plain HTTP fetch holds a request object, a response body and whatever your parse produces — a few megabytes at most, and tens of milliseconds of CPU. A browser context is a different order of magnitude: it downloads CSS, fonts, images and the JavaScript bundle, parses and executes all of it, builds a DOM and a layout tree, and holds all of that until you close it. Indicatively, expect a rendered page to cost somewhere in the region of a hundred times the memory and roughly ten times the wall-clock of the equivalent JSON call, which means a machine that comfortably runs a hundred concurrent HTTP requests will run perhaps four to eight concurrent browser contexts. Those are rules of thumb for planning capacity, not measurements — profile your own pages before sizing a fleet.

That ratio is the argument for the hybrid. Rendering 10% of a crawl roughly doubles its total cost; rendering 100% multiplies it. It is also the argument for a third pattern worth knowing: render once to harvest a signed token or a session cookie, then replay cheap HTTP requests carrying it until it expires. That gives you browser-grade credentials at API-grade cost, and works whenever the signature is attached to a session rather than computed per request.

You can also cut rendering cost directly by refusing to download what you are not going to read. A route handler that aborts images, media, fonts and stylesheets routinely halves page load time on a media-heavy site and cuts memory noticeably, because the browser never decodes any of it.

# blocking_spider.py — abort assets on the rendered path only
from collections.abc import Iterator

import scrapy
from playwright.async_api import Route

BLOCKED = {"image", "media", "font", "stylesheet"}


async def block_assets(route: Route) -> None:
    if route.request.resource_type in BLOCKED:
        await route.abort()
    else:
        await route.continue_()


class LeanRenderSpider(scrapy.Spider):
    name = "lean_render"
    custom_settings = {
        "TWISTED_REACTOR": "twisted.internet.asyncioreactor.AsyncioSelectorReactor",
        "DOWNLOAD_HANDLERS": {
            "https": "scrapy_playwright.handler.ScrapyPlaywrightDownloadHandler",
        },
        "PLAYWRIGHT_MAX_CONTEXTS": 4,
    }

    def start_requests(self) -> Iterator[scrapy.Request]:
        yield scrapy.Request(
            "https://example.com/dashboard",
            meta={"playwright": True, "playwright_page_methods": [
                {"method": "route", "args": ["**/*", block_assets]},
                {"method": "wait_for_selector", "args": ["div.report-row"]},
            ]},
            callback=self.parse,
        )

    def parse(self, response: scrapy.http.Response) -> Iterator[dict]:
        for row in response.css("div.report-row"):
            yield {"label": row.css("::attr(data-label)").get()}

Be aware that blocking stylesheets changes layout, so anything you extract via a visibility check or an element's computed position will behave differently. If your wait condition depends on layout rather than on the presence of a node, keep stylesheets and block only images and media.

Edge Cases and Caveats

Check for the API first, every time. Rendering to obtain data that is available as clean JSON is the single most common waste in SPA scraping. The Network tab is the first stop, not the fallback.
TWISTED_REACTOR must be the asyncio selector reactor. Without it scrapy-playwright fails to initialise, usually with an error about the installed reactor. It is the most frequent setup mistake, and it must be set before the reactor is installed — in settings, not at runtime.
Browsers do not scale linearly. Each context holds real memory, and an unbounded PLAYWRIGHT_MAX_CONTEXTS will exhaust a modest server long before Scrapy's concurrency limit is reached.
wait_until="networkidle" is unreliable on SPAs. Apps with polling or analytics beacons never go idle, so the wait times out. Wait for a specific selector that only exists once the data has rendered.
Private endpoints change without notice. A schema or auth change can arrive overnight. Alert on zero-result runs rather than trusting a green exit code — see Detecting Silent Scraper Failures.
Rendering does not license aggression. A browser makes each request more expensive for the target, not less. Keep DOWNLOAD_DELAY and concurrency polite on the rendered path especially.
GraphQL is a different shape of the same idea. If the Fetch/XHR panel shows a single /graphql endpoint rather than REST paths, the technique is the same but the pagination is cursor-based — see Handling GraphQL Pagination and Cursors.

Frequently Asked Questions

Do I always need Playwright to scrape a single-page app? Usually the opposite. Most single-page apps load their content from a background JSON endpoint that you can call directly with Scrapy, which is faster, cheaper and more stable than rendering. Reach for a browser only when the data depends on client-side execution you cannot replay, such as request signing, canvas output, or a token minted in the page.

What is scrapy-playwright and when should I use it? It is a Scrapy download handler that routes selected requests through a Playwright browser while everything else uses the normal HTTP path. Use it when a minority of pages genuinely need JavaScript execution but you still want one crawl with Scrapy's scheduling, retries, throttling and item pipelines covering the whole job.

How do I find the hidden API a single-page app uses? Open developer tools, go to the Network tab, filter to Fetch or XHR, and reload the page. Look for responses containing JSON that matches what is on screen, then copy the request as cURL so you capture its headers and query parameters exactly, and reproduce it in your spider. Strip the headers back one at a time to find which ones the endpoint actually checks.

Is Playwright slower than Scrapy for the same site? Yes, by a wide margin. A browser downloads and executes the whole page — assets, bundle, layout — where a spider issues one request and reads a JSON body, so the rendered path costs roughly an order of magnitude more time and around a hundred times the memory per page. When both approaches can reach the same data, the direct call wins on speed, cost and reliability.

Web Scraping with Scrapy — the parent topic on spiders, settings and crawl structure.
Reverse-Engineering Private APIs — the discipline behind finding and replaying the endpoint.
Using Playwright for Modern Web Automation — the browser side in depth, for when rendering is unavoidable.
Scrapy vs BeautifulSoup: Which to Use — the prior decision about whether you need the framework at all.