Deploying Scrapers to the Cloud

A scraper that runs on a laptop is a prototype, and this guide — part of Scaling Python Web Scrapers — covers turning it into something that runs unattended without you watching it. The scope is the deployment decision and its consequences: which runtime fits which crawl shape, how secrets and egress addresses are handled, and what a scheduled run must do differently from an interactive one.

Two things break when a scraper moves to the cloud, and neither is the code. The first is the network identity: your home connection looks like a person, and a cloud provider's published address ranges look like exactly what they are. The second is state: a laptop has a filesystem that persists between runs, and most cloud runtimes do not. Almost every "it worked locally" failure traces back to one of those two.

When to Use Each Deployment Pattern

The right home for a scraper depends on how long a run takes, how often it happens, and how much memory the heaviest step needs.

• Serverless functions (AWS Lambda, Cloud Functions) — best for short, bursty jobs that finish well inside the platform's ceiling. Zero cost when idle, instant scale-out, but hard limits on run time, package size and memory. The specific constraints are in Running Scrapers on AWS Lambda.
• Containers (Cloud Run, ECS, Kubernetes) — the general-purpose default. Reproducible environments, headless browsers work, long crawls are fine, and they scale horizontally behind a queue. Scale-to-zero platforms keep the idle cost at nothing.
• Virtual machines — a persistent box you fully control. Right for stateful, always-on crawlers, for anything needing a stable long-lived address, or when you want a local disk that survives. You own patching, monitoring and uptime.
• Managed schedulers and CI — for periodic jobs, a scheduled workflow can build, run and export with no server at all. Cheap and simple up to the point where run time or concurrency exceeds what the runner allows; see Scheduling Scrapers with Cron and GitHub Actions.

A distributed crawl usually combines several of these: a queue such as Celery and Redis with containerised workers, a scheduled producer, and a managed database as the sink.

The decision usually collapses to two questions. Does a single run finish in under five minutes, reliably, including cold start? If not, serverless is out. Does the run need a browser? If so, budget a gigabyte of memory and around 400 MB of image, which pushes you toward containers.

Prerequisites

Docker on your machine, and a cloud account with a container registry. The examples use Python 3.12.

docker --version
pip install "httpx==0.27.2" "selectolax==0.3.21" "structlog==24.4.0"

Step-by-Step: Getting a Scraper Running Unattended

1. Containerise Before Choosing a Platform

Whatever the target, package the scraper as an image first. A container pins the Python version, the system libraries, and any browser binary, so the code that passed locally runs identically in the cloud. Serverless, container services and VMs can all run the same artefact.

# Dockerfile
FROM python:3.12-slim AS base

ENV PYTHONUNBUFFERED=1 \
    PYTHONDONTWRITEBYTECODE=1 \
    PIP_NO_CACHE_DIR=1

WORKDIR /app

COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt

COPY scraper/ ./scraper/

RUN useradd --create-home --uid 10001 scraper
USER scraper

ENTRYPOINT ["python", "-m", "scraper.run"]

PYTHONUNBUFFERED=1 is not cosmetic. Without it, Python buffers stdout when it is not a terminal, so a container that is killed loses every log line it had written — which is precisely the run you most need the logs from.

Copying requirements.txt before the source code means the dependency layer is cached and only rebuilds when the dependencies change, cutting a typical rebuild from 90 seconds to under five.

# scraper/run.py
import os
import sys

import httpx
from selectolax.parser import HTMLParser

HEADERS = {
    "User-Agent": (
        "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 "
        "(KHTML, like Gecko) Chrome/127.0.0.0 Safari/537.36"
    ),
    "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
}


def scrape(target: str) -> list[dict[str, str]]:
    with httpx.Client(timeout=20, follow_redirects=True, headers=HEADERS) as client:
        response = client.get(target)
    response.raise_for_status()

    tree = HTMLParser(response.text)
    rows: list[dict[str, str]] = []
    for card in tree.css("article.product_pod"):
        title = card.css_first("h3 a")
        price = card.css_first("p.price_color")
        rows.append(
            {
                "title": title.attributes.get("title", "") if title else "",
                "price": price.text(strip=True) if price else "",
            }
        )
    return rows


def main() -> int:
    target = os.environ.get("TARGET_URL", "https://books.toscrape.com/")
    rows = scrape(target)
    if not rows:
        print(f"no rows extracted from {target}", file=sys.stderr)
        return 1
    print(f"extracted {len(rows)} rows from {target}")
    return 0


if __name__ == "__main__":
    raise SystemExit(main())

Returning a non-zero exit code when nothing was extracted is what makes the run visible to a scheduler. A scraper that fetches a block page, finds no products and exits 0 will be reported as a successful run forever.

Build and run locally exactly as the cloud will:

docker build -t my-scraper:latest .
docker run --rm -e TARGET_URL="https://books.toscrape.com/" my-scraper:latest

2. Inject Secrets at Run Time

Proxy credentials, API keys and database passwords must never be in the image or the repository. An image layer is a tarball anyone who can pull it can read, and docker history will show a secret passed as a build argument. Read them from the environment at start-up, populated by the platform's secret manager.

# scraper/config.py
import os
from dataclasses import dataclass


class MissingSecret(RuntimeError):
    pass


def require(name: str) -> str:
    value = os.environ.get(name)
    if not value:
        raise MissingSecret(f"required environment variable {name} is not set")
    return value


@dataclass(frozen=True)
class Settings:
    proxy_url: str
    database_url: str
    max_pages: int

    @classmethod
    def from_env(cls) -> "Settings":
        return cls(
            proxy_url=require("PROXY_URL"),
            database_url=require("DATABASE_URL"),
            max_pages=int(os.environ.get("MAX_PAGES", "100")),
        )


if __name__ == "__main__":
    settings = Settings.from_env()
    print(f"configured for up to {settings.max_pages} pages")

Validating everything at start-up rather than at first use is the point. A scraper that discovers its proxy variable is empty on page 4,000 has already sent 4,000 requests from the wrong address and probably burned the target's tolerance for the whole fleet.

3. Plan the Egress Address Before the First Run

This is the deployment concern unique to scraping. Cloud providers publish their address ranges, and anti-bot vendors subscribe to those lists. A datacenter address is not merely suspicious — it is often an outright block rule applied before any behavioural analysis.

The consequences differ by runtime. A serverless function draws from a large shared pool you do not control, so your requests arrive from constantly changing addresses that other tenants may already have poisoned. A VM has one stable address, which is clean until it is noticed and then permanently dirty. Neither is a strategy; routing through proxies is.

# scraper/fetch.py
import httpx

from .config import Settings

HEADERS = {
    "User-Agent": (
        "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 "
        "(KHTML, like Gecko) Chrome/127.0.0.0 Safari/537.36"
    ),
    "Accept-Language": "en-GB,en;q=0.9",
}


def build_client(settings: Settings) -> httpx.Client:
    return httpx.Client(
        proxy=settings.proxy_url,
        headers=HEADERS,
        timeout=httpx.Timeout(connect=10.0, read=25.0, write=10.0, pool=30.0),
        follow_redirects=True,
    )


def check_egress(client: httpx.Client) -> str:
    """Confirm requests leave through the proxy before the crawl starts."""
    response = client.get("https://httpbin.org/ip")
    response.raise_for_status()
    return response.json()["origin"]

Running check_egress as the first action of every deployment is a two-second insurance policy: it fails loudly on a misconfigured proxy rather than quietly crawling from the container's own address. Provider selection and the residential-versus-datacenter trade-off are covered in Residential vs Datacenter Proxies, and rotation strategy in Rotating Proxies and Managing IP Blocks.

4. Make the Run Survive Its Own Time Limit

Every managed runtime will eventually kill a run. Serverless platforms enforce a hard ceiling; container schedulers stop tasks during a deploy; spot instances are reclaimed with two minutes' notice. A run that only writes results at the end loses everything each time.

Two properties fix this. Flush results continuously, so the work already done is durable. And handle SIGTERM, so the grace period before SIGKILL is spent shutting down cleanly instead of being ignored.

# scraper/lifecycle.py
import signal
import sys
import time
from types import FrameType

_shutdown = False


def _on_terminate(signum: int, frame: FrameType | None) -> None:
    global _shutdown
    _shutdown = True
    print(f"received signal {signum}, finishing current page", file=sys.stderr)


def install_handlers() -> None:
    signal.signal(signal.SIGTERM, _on_terminate)
    signal.signal(signal.SIGINT, _on_terminate)


def should_stop(deadline: float) -> bool:
    return _shutdown or time.monotonic() > deadline


def run_until(budget_seconds: float, urls: list[str]) -> int:
    install_handlers()
    deadline = time.monotonic() + budget_seconds
    done = 0
    for url in urls:
        if should_stop(deadline):
            print(f"stopping early after {done} of {len(urls)} URLs", file=sys.stderr)
            break
        done += 1
    return done

Giving the process its own deadline, set slightly below the platform's, is what turns a hard kill into a graceful stop. On a 900-second Lambda ceiling, a 780-second internal budget leaves two minutes to flush results and record where the crawl reached — so the next invocation resumes rather than restarts.

Cold start is the other half of the arithmetic. A slim Python image starts in one to three seconds; one carrying Chromium takes fifteen to thirty. At a five-minute crawl that overhead is noise. At a twenty-second crawl invoked every minute, it is most of your bill.

5. Write Results Somewhere That Outlives the Run

Container filesystems are ephemeral. Anything written to local disk disappears when the task ends, and on a serverless platform /tmp may or may not persist between invocations depending on whether the execution environment is reused — which is worse than not persisting, because it works in testing and fails in production.

The rule is that the only durable outputs are a managed database, an object store, or a message on a queue. Batching matters here too: writing each record individually to object storage costs a request per record, so accumulate a few hundred and upload once. Sink choice, batching and idempotent upserts are covered in Storing and Exporting Scraped Data.

Write a run manifest alongside the data — start time, end time, pages fetched, rows written, exit reason. It costs one small object per run and turns "did last night's crawl work?" into a query rather than a log search.

6. Make Failures Visible

An unattended scraper that fails silently is worse than one that does not run, because you keep making decisions on stale data. Structured logs plus a handful of counters are enough.

# scraper/observability.py
import time

import structlog

structlog.configure(
    processors=[
        structlog.processors.add_log_level,
        structlog.processors.TimeStamper(fmt="iso"),
        structlog.processors.JSONRenderer(),
    ]
)

log = structlog.get_logger()


def run_with_summary(urls: list[str]) -> dict[str, int | float]:
    started = time.monotonic()
    ok = 0
    failed = 0
    for url in urls:
        try:
            log.info("fetch.start", url=url)
            ok += 1
        except Exception as exc:  # noqa: BLE001 - summary counter, re-raised below
            failed += 1
            log.error("fetch.failed", url=url, error=type(exc).__name__)
    summary = {
        "pages_ok": ok,
        "pages_failed": failed,
        "duration_s": round(time.monotonic() - started, 2),
        "success_rate": round(ok / max(len(urls), 1), 4),
    }
    log.info("run.finished", **summary)
    return summary

JSON lines are what every managed log platform can filter on without a custom parser, which turns "alert when success rate drops below 0.9" into a one-line query. The full treatment is in Structured Logging for Python Scrapers, and the alerting rules worth setting in Monitoring and Alerting for Scrapers.

Performance and Scaling Considerations

Match the billing model to the duty cycle. Serverless bills per invocation and per gigabyte-second: cheap for a job that runs for twenty seconds every hour, expensive for one that runs for ten minutes continuously. A VM bills for uptime whether or not it is working, so an always-on box for an hourly crawl wastes most of its cost. Scale-to-zero containers charge while running and nothing while idle, which fits the majority of scheduled crawls.

Image size is start-up latency. A python:3.12-slim base with a few pure-Python dependencies produces an image around 150 MB that pulls in a couple of seconds. Adding Playwright and Chromium takes it past 1.5 GB and adds tens of seconds to every cold start. If only some pages need a browser, split into two images and route accordingly rather than paying the browser cost on every request.

Memory sizing changes CPU on serverless. On Lambda and Cloud Functions, CPU is allocated in proportion to configured memory, so a function at 512 MB gets roughly a third of the CPU of one at 1,536 MB. A parsing-heavy scraper is often cheaper at higher memory because it finishes proportionally faster — measure both, since the billing is memory multiplied by duration.

Concurrency is a target-side limit, not a platform limit. A container platform will happily run two hundred replicas. The site will not happily serve two hundred concurrent clients from one organisation. Enforce a per-domain rate limit in shared state, not per replica, or the fleet's effective rate is your intended limit times the replica count.

Prefer many short runs over one long one. A crawl split into hourly slices of a thousand URLs recovers from a failure by losing an hour, resumes trivially, and fits inside every platform's time limit. One nightly six-hour run fails as a unit.

Pin dependency versions and rebuild deliberately. An unpinned requirements.txt means the image you deploy on Friday is not the one you tested on Monday, and a Scrapy or Playwright minor release can change behaviour. Pin exact versions and upgrade as an explicit change — the framework-specific settings that go with it are in Web Scraping with Scrapy.

Common Errors and Fixes

KeyError: 'PROXY_URL' in the cloud but not locally The variable is in your shell or a local .env but was never configured on the platform. Read configuration through a helper that raises a named error at start-up, and add the variable to the task definition or function configuration — not to the image.

Task timed out after 900.00 seconds The run exceeded the platform's hard limit and was killed mid-crawl with nothing flushed. Give the process an internal deadline below the platform's, persist progress continuously, and split the URL list so each invocation handles a slice.

[Errno 28] No space left on device The runtime's writable layer or /tmp filled up, commonly from cached HTML, browser profiles or downloaded files. Serverless /tmp is typically 512 MB by default. Stream results to durable storage and delete temporary files as you go rather than accumulating them.

playwright._impl._errors.Error: Executable doesn't exist at /ms-playwright/chromium-... The browser binaries were not installed in the image. Add RUN playwright install --with-deps chromium after installing the package, and confirm the install runs as the same user that runs the scraper — installing as root and running as a non-root user puts the binaries somewhere unreadable.

Every request returns 403 from the cloud, 200 from your laptop The target is blocking the provider's address range. Verify with check_egress that requests actually leave through your proxy, and confirm the proxy is applied to the client that performs the crawl rather than only to a test client.

The scheduled job reports success but produces no data The process exited 0 after extracting nothing, usually because a block page parsed cleanly and matched no selectors. Return a non-zero exit code when the row count is below an expected floor, and alert on rows written rather than on process exit alone.

OSError: [Errno 99] Cannot assign requested address under high concurrency The container ran out of ephemeral ports because connections are being opened and closed rather than pooled. Reuse a single client with keep-alive enabled, and lower concurrency until the socket count is stable.

Logs stop abruptly with no error Output was buffered and lost when the process was killed. Set PYTHONUNBUFFERED=1 in the image, write logs to stdout rather than a file, and flush after each record if the platform batches log delivery.

Frequently Asked Questions

Serverless or containers for a scraper? Serverless suits short, spiky jobs that finish well inside the time limit and need no persistent state. Containers suit long crawls, headless browsers and anything holding state between pages. When in doubt, containerise: the same image runs on a scheduler, a queue worker or a VM, so the choice stays reversible.

Why do cloud scrapers get blocked when the same code works locally? Your home connection looks residential, and cloud egress ranges are published and widely blocklisted. The code is identical; the network identity is not. Route requests through residential or rotating proxies and verify the egress address at start-up.

How do I keep a long crawl within a platform's time limit? Split the work. Give each invocation a slice of the URL list, persist a cursor recording where it stopped, and have the next invocation resume from it. This also makes the crawl restartable after any failure, not just a timeout.

Where should a cloud scraper store its data? In managed durable storage — a hosted database or an object store — never the runtime's local disk, which vanishes when the task ends. Write incrementally so a killed run keeps whatever it had already collected.

Do I need Kubernetes for this? Rarely. A scheduled container service plus a queue covers almost every scraping workload with far less to operate. Kubernetes starts to earn its complexity when you are running many different crawlers with distinct resource profiles and already have both the platform and the people to run it.

Related

• Scaling Python Web Scrapers — how deployment fits alongside concurrency and distribution
• Running Scrapers on AWS Lambda — the specific serverless limits and packaging
• Scheduling Scrapers with Cron and GitHub Actions — periodic runs without a server
• Monitoring and Alerting for Scrapers — knowing when an unattended run has failed
• Storing and Exporting Scraped Data — durable sinks for ephemeral runtimes