Handling GraphQL Pagination and Cursors

This walkthrough extends Scraping GraphQL Endpoints with Python with the one piece every real list query needs: following cursor-based pagination to fetch every record rather than just the first page.

Each page returns an endCursor that feeds the next request's after variable; the loop ends when hasNextPage is false.

GraphQL connections paginate with an opaque cursor, not a page number. You request pageInfo { endCursor hasNextPage } alongside your data, pass the returned endCursor back into the query's after variable, and repeat while hasNextPage is true. Start with after: null for the first page, stop the moment hasNextPage turns false — and also stop if a page returns zero edges, because some servers report one page too many.

Why Cursors Instead of Page Numbers

Offset pagination breaks when the underlying list changes between requests. Insert a row near the top and every later page shifts by one, so you receive one record twice and never see another. On a catalogue that is being edited while you crawl, that is not a rare edge case — it happens on most long runs.

Insert one row at the head of the list between two page requests and offset pagination loses a record it never saw. A cursor names a position in the result set, so the same insertion changes nothing.

A relay-style connection solves this with a cursor: an opaque token that marks a fixed position in the result set. Because the cursor identifies a specific record rather than a numeric distance from the start, the sequence stays stable even as data changes underneath you. Deletions are handled the same way — if the record a cursor points at is removed, a conforming server still resolves the position rather than erroring. This is the same reliability problem that offset-based pagination and infinite scroll faces on rendered pages, solved properly at the API layer.

The connection shape is consistent across every relay-compliant server:

edges — a list where each entry wraps a node (your record) and that record's own cursor.
pageInfo — metadata containing at least endCursor, hasNextPage, and, for backward paging, startCursor and hasPreviousPage.

Your query asks for the fields you want on each node, plus pageInfo, and takes two arguments: first (records per page) and after (the cursor to start after).

PAGINATED_QUERY = """
query GetProducts($first: Int!, $after: String) {
  products(first: $first, after: $after) {
    edges {
      cursor
      node { id name price sku }
    }
    pageInfo { endCursor hasNextPage }
  }
}
"""

$first is Int! — non-null, so it must always be present. $after is plain String, nullable, which is what lets you pass None on the first call. Declaring $after as String! is a common mistake and produces Variable "$after" of non-null type "String!" must not be null on the very first request.

The Full Loop in Python

The loop is small once the shape is clear: run the query, collect the nodes, read pageInfo, and either continue with the new cursor or stop.

The loop carries exactly one piece of state between iterations: the cursor. Both exit conditions must be checked, because a server that reports hasNextPage one page too long will otherwise spin forever.

import httpx

GRAPHQL_URL = "https://api.example.com/graphql"

PAGINATED_QUERY = """
query GetProducts($first: Int!, $after: String) {
  products(first: $first, after: $after) {
    edges { node { id name price sku } }
    pageInfo { endCursor hasNextPage }
  }
}
"""


def run_query(client: httpx.Client, variables: dict) -> dict:
    response = client.post(
        GRAPHQL_URL,
        json={"query": PAGINATED_QUERY, "variables": variables,
              "operationName": "GetProducts"},
    )
    response.raise_for_status()
    result = response.json()
    if result.get("errors"):
        raise RuntimeError(result["errors"])
    return result["data"]["products"]


def fetch_all_products(page_size: int = 50, max_pages: int = 1000) -> list[dict]:
    headers = {
        "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 "
                      "(KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36",
        "Accept": "application/json",
        "Content-Type": "application/json",
    }
    products: list[dict] = []
    seen: set[str] = set()
    cursor: str | None = None
    with httpx.Client(http2=True, timeout=20, headers=headers) as client:
        for page in range(max_pages):
            connection = run_query(client, {"first": page_size, "after": cursor})
            edges = connection["edges"]
            if not edges:                       # server over-reported hasNextPage
                break
            for edge in edges:
                node = edge["node"]
                if node["id"] in seen:          # defensive: overlapping pages
                    continue
                seen.add(node["id"])
                products.append(node)
            page_info = connection["pageInfo"]
            if not page_info["hasNextPage"]:
                break
            next_cursor = page_info["endCursor"]
            if next_cursor == cursor or next_cursor is None:
                break                           # cursor did not advance; stop
            cursor = next_cursor
        else:
            raise RuntimeError(f"stopped at the {max_pages}-page ceiling")
    return products


if __name__ == "__main__":
    all_products = fetch_all_products()
    print(f"collected {len(all_products)} products")

Four guards do the real work here. The empty-edges check handles servers whose hasNextPage is optimistic. The seen set makes an overlapping page idempotent instead of duplicating records. The cursor-did-not-advance check catches the genuine infinite loop, where a server returns the same endCursor forever. And the for…else clause turns hitting the page ceiling into a loud error rather than a silently truncated dataset — a distinction that matters, because a quiet truncation looks exactly like a small catalogue.

Reusing one Client pools the connection so every page after the first skips the TCP and TLS handshake, which on a 200-page crawl is typically 30–60 seconds saved. Once you have the flat list of nodes, reshape it using the patterns in Parsing JSON and XML Responses and check the field types with Cleaning and Validating Scraped Data before it reaches storage.

Making a Sequential Loop Faster

Cursor pagination cannot be parallelised directly — each page's cursor is only known after the previous response arrives. What you can parallelise is several independent cursor streams. Split the collection along a dimension the API already filters on, then page each partition concurrently.

import asyncio
import httpx

PARTITIONED_QUERY = """
query ByCategory($slug: String!, $first: Int!, $after: String) {
  products(category: $slug, first: $first, after: $after) {
    edges { node { id name price } }
    pageInfo { endCursor hasNextPage }
  }
}
"""


async def page_category(client: httpx.AsyncClient, sem: asyncio.Semaphore,
                        slug: str, page_size: int = 50) -> list[dict]:
    nodes: list[dict] = []
    cursor: str | None = None
    while True:
        async with sem:
            response = await client.post(
                "https://api.example.com/graphql",
                json={"query": PARTITIONED_QUERY,
                      "variables": {"slug": slug, "first": page_size, "after": cursor}},
            )
        response.raise_for_status()
        connection = response.json()["data"]["products"]
        if not connection["edges"]:
            return nodes
        nodes.extend(edge["node"] for edge in connection["edges"])
        if not connection["pageInfo"]["hasNextPage"]:
            return nodes
        cursor = connection["pageInfo"]["endCursor"]


async def main(slugs: list[str]) -> list[dict]:
    headers = {
        "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 "
                      "(KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36",
        "Content-Type": "application/json",
    }
    sem = asyncio.Semaphore(6)
    async with httpx.AsyncClient(http2=True, timeout=20, headers=headers) as client:
        batches = await asyncio.gather(*(page_category(client, sem, s) for s in slugs))
    return [node for batch in batches for node in batch]


if __name__ == "__main__":
    print(len(asyncio.run(main(["chairs", "desks", "lighting"]))))

The semaphore is not optional. Six concurrent streams against a complexity-limited endpoint is comfortable; thirty will trip a cost-per-minute limit and cost you more time in backoff than the parallelism saved. The general bounded-concurrency pattern is covered in Asynchronous Scraping with asyncio and HTTPX.

Resuming an Interrupted Crawl

A 400-page crawl that dies on page 380 should not start again from page 1. Checkpoint the cursor after each successful page and reload it on startup. The state is a single string, so the storage can be a file.

import json
import pathlib

STATE = pathlib.Path("products.state.json")


def load_state() -> tuple[str | None, int]:
    if not STATE.exists():
        return None, 0
    saved = json.loads(STATE.read_text(encoding="utf-8"))
    return saved.get("cursor"), int(saved.get("count", 0))


def save_state(cursor: str | None, count: int) -> None:
    tmp = STATE.with_suffix(".tmp")
    tmp.write_text(json.dumps({"cursor": cursor, "count": count}), encoding="utf-8")
    tmp.replace(STATE)          # atomic: a crash mid-write cannot corrupt the file

Write to a temporary path and rename, because replace is atomic on both POSIX and Windows while a direct overwrite can leave a truncated file if the process dies mid-write. Then append each page's records to your output before saving the cursor, never after — the ordering guarantees you can only ever duplicate a page, never skip one, and duplicates are removable by primary key while gaps are not.

Two caveats apply to resumption specifically. If the server encodes a snapshot identifier in its cursors, a token from yesterday may be rejected with Invalid cursor or, worse, silently reinterpreted against the current data; treat a checkpoint older than a few hours as expired and restart. And if the underlying list is sorted by something mutable, such as popularity or price, resuming from a cursor is only meaningful while the sort is stable — sort by an immutable key such as id or creation time whenever the API offers the choice.

Edge Cases and Caveats

Cursors are opaque — never construct one. A cursor is usually base64 of something like arrayconnection:49, and the encoding is an implementation detail that can change without notice. Only ever pass back a value the server gave you.
hasNextPage can be optimistic. Some servers return true on the final page and then an empty edges list on the next call. Break on zero edges regardless of the flag.
Respect the first ceiling. Many APIs cap page size at 100. Requesting first: 1000 either errors with Requested 1000 records, maximum is 100 or silently clamps, which means your loop runs ten times as many pages as you expected.
Partial results still carry a cursor. If errors is present but data is populated, the connection may still be valid. Decide deliberately whether to keep the partial page and continue, or abort — silently discarding it loses records.
Backward pagination exists. Relay also defines last and before with startCursor. It is occasionally the faster route when you only want the newest records from a long list.
totalCount is not part of the spec. It is a common extension but not guaranteed, and where it exists it is often approximate. Use hasNextPage for loop control and treat totalCount as a sanity check only.
Cursors can expire. Servers that encode a snapshot identifier will reject a cursor from an hour-old crawl with Invalid cursor. Resume from the start rather than persisting cursors across runs.
Per-edge cursor and pageInfo.endCursor are not always interchangeable. The spec says endCursor equals the last edge's cursor, and conforming servers honour that, but implementations that paginate over a materialised view sometimes return a cursor that encodes the page boundary instead. Always use pageInfo.endCursor, which is the value the server intends you to send back.
Filters and cursors interact. A cursor is only valid for the argument set that produced it. Changing orderBy, a date filter or a category between pages either errors or, worse, silently resumes in a different ordering — build the whole variable set once and change only after.
Pace long crawls. Thousands of sequential pages add up in both time and cost points; monitor the run and alert if the page count changes sharply between days, as covered in Detecting Silent Scraper Failures.

Frequently Asked Questions

What exactly is a cursor? An opaque token marking a record's position in the result set. Treat it as a black box: read it from pageInfo.endCursor and pass it straight back as the after variable. Its internal format is the server's business and may change between deployments, so never parse or generate one yourself.

How do I know when to stop paginating? Loop while pageInfo.hasNextPage is true, feeding endCursor into after each time. Add two safety nets: stop if a page returns zero edges, and stop if endCursor comes back unchanged, since both indicate a server that will otherwise keep you looping indefinitely.

Can I paginate faster with concurrent requests? Not within a single stream — each page's cursor is only known after the previous response. Parallelise by splitting the work along another dimension such as category, region or date range, and page each of those streams concurrently behind a semaphore.

The response has no pageInfo. How do I page then? That endpoint uses a non-relay style, most often limit/offset arguments or a bare nextToken field echoed back on the next call. Inspect a real request from the site's own frontend to see its exact pagination arguments, using the discovery method in Reverse-Engineering Private APIs in Python.

Scraping GraphQL Endpoints with Python — the parent topic, covering schema discovery and query building
Parsing JSON and XML Responses — reshaping the collected nodes into records
Flattening Nested JSON with pandas — turning nested nodes into a table
Extracting JSON-LD and Structured Data — the fallback when there is no API to page