Running Scrapers on AWS Lambda

Lambda runs a function on demand, charges nothing while idle and scales out without a scheduler of your own, which makes it a natural home for periodic scrapers — and this page, part of Deploying Scrapers to the Cloud, covers the ceilings that shape how you have to build for it.

EventBridge invokes a Lambda function on a schedule; the function uses a dependency layer and writes results to S3.

Lambda is an excellent fit for HTTP scrapers whose unit of work finishes in seconds and whose output goes straight to S3 or a database. Ship dependencies as a layer for anything under 250 MB unzipped and as a container image above that, keep every invocation comfortably under the 15-minute hard limit by making one invocation handle one page or one small batch, and drive the schedule from EventBridge. Full headless browsers run but fight the constraints at every step; if browser rendering is the main job, a container service is the better home. The cost crossover against a small always-on VM lands somewhere around a few hours of compute per day.

The Ceilings You Have to Design Around

Nothing about Lambda's model is negotiable, so it is worth internalising the four numbers before writing any code.

Four fixed ceilings decide how a Lambda scraper has to be shaped: short runs, memory that buys CPU, scratch space that vanishes, and a package budget that pushes browsers into a container image.

The 15-minute timeout is the one that reshapes architecture. A crawl that loops over 5,000 URLs in one function will be killed part-way through with Task timed out after 900.00 seconds, and everything held in memory is lost. The fix is not a bigger timeout — it is a smaller unit of work.

Memory is also your CPU dial. Lambda allocates vCPU in proportion to configured memory, crossing roughly one full vCPU at about 1,769 MB. A parsing-heavy function set to 256 MB is often more expensive than the same function at 1,024 MB, because it runs more than four times as long for a quarter of the per-millisecond price. Measure both settings before assuming the small one is cheaper.

Only /tmp is writable, it starts at 512 MB, and it is gone when the execution environment is recycled. It is scratch space for a download you are about to upload, never storage.

Package size decides your packaging strategy, which is the next section.

Packaging Dependencies

The Lambda Python runtime ships the standard library and boto3, and nothing else. Everything your scraper imports has to travel with it, built for the runtime's platform — Amazon Linux, x86_64 or arm64 depending on your architecture setting. Installing on a Mac and zipping the result is the classic way to produce Unable to import module 'handler': No module named '_cffi_backend' at runtime.

Measure the unzipped size of your dependency tree first — that single number decides whether you ship a plain zip, publish a layer, or move the whole function into a container image.

For the middle band, a layer is a zip of dependencies mounted read-only at /opt/python, which is already on sys.path. Keeping it separate from your handler means you can redeploy logic in a two-kilobyte upload instead of rebuilding the whole bundle.

mkdir -p layer/python
pip install httpx==0.27.0 selectolax==0.3.21 -t layer/python \
  --platform manylinux2014_x86_64 --only-binary=:all: --python-version 3.12
cd layer && zip -qr ../scraper-layer.zip python && cd ..
du -sh layer/python   # check you are under 250 MB unzipped

Publish it and note the returned version ARN — layers are immutable, so every rebuild produces a new version number that you attach explicitly:

aws lambda publish-layer-version \
  --layer-name scraper-deps \
  --zip-file fileb://scraper-layer.zip \
  --compatible-runtimes python3.12 \
  --compatible-architectures x86_64

The Handler

A handler receives (event, context). Read the target from the event so one deployed function can serve many schedules and queue messages, send an explicit User-Agent, and write results somewhere durable before returning — anything left on disk or in memory is gone.

# handler.py
import json
import os

import boto3
import httpx

HEADERS = {
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 "
    "(KHTML, like Gecko) Chrome/125.0 Safari/537.36",
    "Accept": "text/html,application/xhtml+xml",
}

# Module scope: reused across warm invocations, so the pool and the TLS
# session survive between calls in the same execution environment.
CLIENT = httpx.Client(timeout=10.0, follow_redirects=True, headers=HEADERS)
S3 = boto3.client("s3")


def lambda_handler(event: dict, context: object) -> dict:
    url = event.get("url", "https://books.toscrape.com/")
    remaining_ms = context.get_remaining_time_in_millis()
    if remaining_ms < 3000:
        return {"statusCode": 503, "body": json.dumps({"skipped": url})}

    response = CLIENT.get(url)
    response.raise_for_status()

    bucket = os.environ["RESULT_BUCKET"]
    key = f"pages/{context.aws_request_id}.html"
    S3.put_object(Bucket=bucket, Key=key, Body=response.content)

    return {
        "statusCode": 200,
        "body": json.dumps({"url": url, "bytes": len(response.content), "saved": key}),
    }

Two habits in that snippet pay for themselves. Constructing the httpx.Client and the boto3 client at module scope means warm invocations reuse the connection pool instead of redoing DNS and TLS every time — on a function invoked once a second, that is most of the latency. And reading context.get_remaining_time_in_millis() lets the function bail out cleanly instead of being killed mid-write; when work is fed from a queue, returning without deleting the message means it will be redelivered rather than lost.

Staying Inside the Timeout

The rule is one invocation, one small unit of work. Instead of a spider that walks a site, deploy a function that fetches a single URL and writes its output, and let something else supply the URLs: EventBridge for a fixed list, SQS for a discovered frontier, or a Step Functions map for a fan-out with a known shape. Lambda's concurrency then does the parallelism that a worker pool would do in Distributed Crawling with Celery and Redis, with the queue instead of a broker you run.

The property this buys you is retry safety. A short, independent invocation that writes an idempotent record can be replayed at no cost, which matters because Lambda retries asynchronous invocations twice by default and SQS redelivers on visibility timeout. Make the write idempotent — an ON CONFLICT upsert as in Saving Scraped Data to PostgreSQL, or an S3 key derived from the URL rather than the request ID — and duplicate deliveries stop mattering.

Headless Browsers

Chromium plus its shared libraries is far past the layer budget, so the only realistic route is a container image, and even then the sandbox needs coaxing.

# browser_handler.py — inside a container image with chromium bundled
from playwright.sync_api import sync_playwright

UA = (
    "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 "
    "(KHTML, like Gecko) Chrome/125.0 Safari/537.36"
)


def lambda_handler(event: dict, context: object) -> dict:
    url = event.get("url", "https://books.toscrape.com/")
    with sync_playwright() as p:
        browser = p.chromium.launch(
            args=[
                "--no-sandbox",             # Lambda has no user namespaces
                "--single-process",         # the sandbox dislikes zygote processes
                "--disable-dev-shm-usage",  # /dev/shm is 64 MB here
                "--disable-gpu",
            ]
        )
        page = browser.new_page(user_agent=UA)
        page.goto(url, wait_until="networkidle", timeout=20000)
        title = page.title()
        browser.close()
    return {"statusCode": 200, "body": title}

Budget at least 2,048 MB of memory for a single Chromium page, and expect a cold start of several seconds while the image layers are pulled and the browser boots — versus a couple of hundred milliseconds for a plain zip. If you are launching a browser on most invocations, you are paying container-service prices for serverless ergonomics; compare with the options in Using Playwright for Modern Web Automation before committing.

Scheduling with EventBridge

An EventBridge rule invokes the function on a cron or rate expression with no server involved. Note that the AWS cron syntax has six fields, not five, and that the day-of-week and day-of-month fields cannot both be * — one must be ?.

aws events put-rule \
  --name nightly-scrape \
  --schedule-expression "cron(0 3 * * ? *)"

aws events put-targets \
  --rule nightly-scrape \
  --targets 'Id=1,Arn=arn:aws:lambda:us-east-1:123456789012:function:scraper,Input={"url":"https://books.toscrape.com/"}'

aws lambda add-permission \
  --function-name scraper \
  --statement-id nightly-scrape \
  --action lambda:InvokeFunction \
  --principal events.amazonaws.com

The add-permission call is the step people forget: without it the rule fires and silently fails, because EventBridge is not authorised to invoke the function. If you would rather keep the schedule in a repository next to the code, Scheduling Scrapers with Cron and GitHub Actions covers the alternative.

Working Out Whether It Is Cheaper

Lambda bills per request and per gigabyte-second of allocated memory. Using published on-demand rates for x86 in us-east-1 at the time of writing — roughly $0.20 per million requests and $0.0000166667 per GB-second, both of which vary by region and change over time — a 1,024 MB function running 4 seconds per invocation costs about $0.000067 per call.

That gives a usable rule of thumb. At 1,000 invocations a day the bill is around $2 a month, and much of it may fall inside the free tier. At 100,000 invocations a day it is around $200 a month, at which point a small always-on instance running the same work as a queue consumer is several times cheaper — a t3.medium on demand is roughly $30 a month before storage. Treat these as indicative arithmetic to run yourself with your own numbers, not as a quote.

The crossover moves in Lambda's favour when the duty cycle is spiky: a job that needs 200 concurrent workers for four minutes each night would need an autoscaling group and a scale-down policy on a VM, and Lambda simply runs it. It moves against Lambda when work is continuous, when each unit is long, or when you need a stable egress IP.

Edge Cases and Caveats

Cold starts scale with package size. A slim zip typically resumes in a couple of hundred milliseconds; a container image with a browser can take seconds. Trim unused dependencies, and use provisioned concurrency only if latency genuinely matters — it bills whether or not the function is called.
Egress IPs are AWS ranges, shared and widely blocked. Many defended sites treat them as hostile by default. Route through proxies from Rotating Proxies and Managing IP Blocks; putting the function in a VPC with a NAT gateway gives a stable IP but also a single address that is easy to block once noticed, and adds NAT data-processing charges.
Module-scope state persists between invocations. That is a feature for connection pools and a bug for anything mutable — a module-level seen_urls set will carry over between unrelated invocations and quietly deduplicate work it should not.
Retries are on by default for async invokes. EventBridge and asynchronous calls retry twice with backoff before hitting the dead-letter destination, so a non-idempotent write can execute three times.
Account concurrency is a shared budget. The default regional limit of 1,000 concurrent executions is shared across every function in the account; a fan-out crawl can throttle unrelated production workloads. Set a reserved concurrency on the scraper.
CloudWatch Logs is a real cost line. Printing every response body at high invocation rates can cost more than the compute. Log structured events at a sane level — see Structured Logging for Python Scrapers.
/tmp is not shared and not persistent. Two concurrent invocations have separate filesystems, and a warm environment may hand you leftover files from the previous call in the same sandbox.

Frequently Asked Questions

Can I run Selenium or Playwright on Lambda? Yes, using a container image that bundles a Lambda-compatible headless Chromium and launching with --no-sandbox, --single-process and --disable-dev-shm-usage. It works, but it needs at least 2 GB of memory, has multi-second cold starts and costs far more per page than an HTTP fetch, so it is worth confirming the site really needs rendering first.

How do I avoid hitting the 15-minute timeout? Stop running the crawl inside one invocation. Make each invocation handle a single page or a small batch, persist its result immediately, and drive many invocations from EventBridge, SQS or Step Functions. Short independent runs also make retries safe, which a long serial crawl never is.

Where should a Lambda scraper store its data? In managed storage outside the function: S3 for raw pages and columnar exports, or a hosted database for structured records. The filesystem is wiped when the execution environment is recycled and is not shared between concurrent invocations, so persist on every run. Storing and Exporting Scraped Data covers choosing the sink.

Is Lambda cheaper than a VM for scraping? For bursty or infrequent work, usually yes, because idle time is free and you never pay for a machine waiting for 3am. For continuous high-volume crawling the per-GB-second charge overtakes a right-sized instance, often somewhere in the region of a few compute-hours per day. Work the arithmetic with your own invocation count, duration and memory setting before deciding.

Deploying Scrapers to the Cloud — the parent topic comparing serverless, containers and virtual machines.
Scheduling Scrapers with Cron and GitHub Actions — the lighter-weight way to run a job on a timetable.
Monitoring and Alerting for Scrapers — turning invocation logs and metrics into something that pages you.
Exporting Scraped Data to CSV and Parquet — the file formats worth writing to S3 from a function.