Scheduling Scrapers with Cron and GitHub Actions

Most scrapers do not need to run continuously, they need to run regularly, and this page — part of Deploying Scrapers to the Cloud — covers the two schedulers that between them handle almost every periodic job: system cron on a machine you own, and a GitHub Actions scheduled workflow on a machine you do not.

Use system cron when you already run an always-on box and want exact control over the environment, the network path and the run time. Use a GitHub Actions schedule when you would rather not own a server at all: you supply a cron expression and a script, GitHub supplies the runner, the Python install, the log retention and the secret storage. For a small periodic scrape the Actions route is less work in total, at the cost of an imprecise start time and a shared, widely-blocked egress IP. Neither is appropriate for continuous high-volume crawling — that is a queue's job.

What You Are Actually Choosing Between

The two options look like a choice of syntax, but the cron expressions are nearly identical. The real difference is how much of the stack stays your responsibility.

With cron, everything below your script is yours: the machine has to stay up, the Python version has to be patched, the log file has to be rotated before it fills the disk, and a failed run is visible only if you arranged for it to be. In exchange you get a stable IP address, no minute limits, no repository-activity rules, and a start time accurate to the second.

With Actions, GitHub owns the timer, the runner image, the Python toolchain and 90 days of run logs, and gives you encrypted secrets and a manual re-run button. In exchange you accept a queued start, a monthly minutes budget on private repositories, and an egress address from a cloud range.

Scheduling with System Cron

A crontab line is five time fields plus a command. The fields are minute, hour, day-of-month, month, day-of-week, and cron runs the command under a minimal environment — no virtualenv activation, often no PATH beyond /usr/bin:/bin, and a working directory of $HOME. Address all three explicitly.

# crontab -e
SHELL=/bin/bash
MAILTO=ops@example.com

# 03:00 every day, in the project directory, using the venv interpreter
0 3 * * * cd /home/scraper/app && /home/scraper/.venv/bin/python run.py >> /home/scraper/logs/scrape.log 2>&1

>> ... 2>&1 is doing real work there: without redirecting stderr, cron mails the output to MAILTO and, if no mailer is configured, drops it entirely — which is how a scraper can be dead for a month without anyone noticing.

The script itself is ordinary Python. Read configuration from the environment, send an explicit User-Agent, and exit non-zero on failure so the exit status carries the signal.

# run.py
import csv
import os
import sys

import httpx
from selectolax.parser import HTMLParser

HEADERS = {
    "User-Agent": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 "
    "(KHTML, like Gecko) Chrome/125.0 Safari/537.36",
    "Accept": "text/html,application/xhtml+xml",
}


def scrape(url: str) -> list[dict[str, str]]:
    with httpx.Client(timeout=15.0, follow_redirects=True, headers=HEADERS) as client:
        response = client.get(url)
    response.raise_for_status()
    tree = HTMLParser(response.text)
    rows: list[dict[str, str]] = []
    for card in tree.css("article.product_pod"):
        title = card.css_first("h3 a")
        price = card.css_first("p.price_color")
        if title is None or price is None:
            continue
        rows.append({"title": title.attributes.get("title", ""), "price": price.text(strip=True)})
    return rows


def main() -> None:
    url = os.environ.get("TARGET_URL", "https://books.toscrape.com/")
    rows = scrape(url)
    if not rows:
        raise SystemExit("no rows extracted; selectors may have broken")
    with open("results.csv", "w", newline="", encoding="utf-8") as fh:
        writer = csv.DictWriter(fh, fieldnames=["title", "price"])
        writer.writeheader()
        writer.writerows(rows)
    print(f"wrote {len(rows)} rows")


if __name__ == "__main__":
    try:
        main()
    except httpx.HTTPError as exc:
        print(f"scrape failed: {exc}", file=sys.stderr)
        sys.exit(1)

The empty-result check matters more than the exception handling. A selector change usually produces a successful HTTP request and zero rows, which cron reports as a clean run — the classic silent failure described in Detecting Silent Scraper Failures.

If you want overlap protection on a slow job, wrap the command in flock, which is available on any modern Linux and costs nothing:

0 * * * * /usr/bin/flock -n /tmp/scrape.lock /home/scraper/.venv/bin/python /home/scraper/app/run.py >> /home/scraper/logs/scrape.log 2>&1

-n makes a second run exit immediately rather than queueing, so a job that overruns its hour skips one cycle instead of doubling up.

Scheduling with GitHub Actions

An on: schedule trigger runs a workflow on GitHub's infrastructure. The cron expression uses the same five fields and is always evaluated in UTC — there is no timezone option, so a "9am local" job needs the offset baked in and adjusted when daylight saving changes.

# .github/workflows/scrape.yml
name: scheduled-scrape

on:
  schedule:
    - cron: "0 3 * * *"        # 03:00 UTC daily
  workflow_dispatch: {}         # manual runs from the Actions tab

concurrency:
  group: scheduled-scrape
  cancel-in-progress: false     # queue a late run instead of overlapping

permissions:
  contents: write               # needed to push results back

jobs:
  scrape:
    runs-on: ubuntu-latest
    timeout-minutes: 20
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.12"
          cache: pip
      - run: pip install httpx==0.27.0 selectolax==0.3.21
      - name: Run scraper
        env:
          TARGET_URL: ${{ secrets.TARGET_URL }}
          PROXY_URL: ${{ secrets.PROXY_URL }}
        run: python run.py
      - name: Commit results
        run: |
          git config user.name "scraper-bot"
          git config user.email "bot@users.noreply.github.com"
          git add results.csv
          git commit -m "data: scheduled scrape $(date -u +%FT%TZ)" || exit 0
          git push

Four lines in that file are the difference between a workflow that works and one that surprises you. workflow_dispatch gives a manual trigger so you can test without waiting for 3am. The concurrency block stops a slow run overlapping the next schedule. timeout-minutes caps a hung job before it eats the minutes budget — the default is six hours. And permissions: contents: write is required for the push, because the default token is read-only on many repositories.

Schedules Drift, and That Is Normal

A hosted cron expression is a request to be queued at that minute, not a promise to start at it. Runs at popular times — the top of the hour, and especially midnight UTC — queue behind everyone else's.

Two practical consequences. First, never build logic that assumes the run started at the stated minute; derive time windows from the data or from date -u inside the job, not from the schedule. Second, avoid 0 * * * * and 0 0 * * * if you can — a schedule of 17 3 * * * typically starts closer to its stated time than 0 3 * * * simply because fewer jobs are queued at that minute.

The other scheduling surprise is specific to GitHub: scheduled workflows are disabled automatically after 60 days without any repository activity. A scraper that commits its results to the same repository keeps itself alive as a side effect; one that pushes to S3 does not, and will quietly stop. Either commit something, or run a lightweight monthly job that does.

Secrets and Proxies

Never put proxy credentials or API keys in the workflow file — it is in the repository, and forks and clones carry it. Store them as encrypted repository secrets under Settings, Secrets and variables, Actions, and reference them as ${{ secrets.NAME }}, which injects them as environment variables and masks them in the log output.

# fetch.py
import os

import httpx

HEADERS = {
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 "
    "(KHTML, like Gecko) Chrome/125.0 Safari/537.36",
}


def fetch(url: str) -> httpx.Response:
    proxy = os.environ.get("PROXY_URL")  # from ${{ secrets.PROXY_URL }}
    with httpx.Client(proxy=proxy, timeout=20.0, headers=HEADERS) as client:
        response = client.get(url)
    response.raise_for_status()
    return response

The masking is textual and only covers exact matches, so a URL-encoded or partially-printed credential slips through. Do not print request objects, do not enable debug logging of headers, and treat Authorization and proxy URLs as things you never log at all. A proxy password leaked into a public run log is visible to anyone, permanently.

Runners also scrape from GitHub's cloud IP ranges, which many defended sites block outright. If the target has any protection, the proxy is not optional — see Rotating Proxies and Managing IP Blocks.

Getting the Output Somewhere Durable

A scheduled run that leaves its data on the runner has achieved nothing; the filesystem is destroyed when the job ends. Three patterns, in increasing order of scale:

• Commit to the repository. For a few hundred kilobytes a run, this gives a free versioned history and a visible diff per scrape, and has the useful side effect of keeping the schedule enabled. It bloats the repository quickly if the file is large or the cadence is high, because every version is stored forever.
• Upload as an artifact. Good for transient output you want to inspect for a few days without keeping it in git history.
• Push to object storage or a database. The right answer for anything sizable, and the only one that scales. Validate before writing so a broken parse cannot poison the store — see Validating Scraped Data with Pydantic.

      - name: Upload artifact
        uses: actions/upload-artifact@v4
        with:
          name: scrape-results
          path: results.csv
          retention-days: 7

For columnar output that a query engine can read directly from a bucket, Exporting Scraped Data to CSV and Parquet covers the format choice.

Edge Cases and Caveats

• Actions cron is UTC only and best effort. There is no timezone field, and start times drift by minutes under load. Anything needing second-level precision needs a scheduler you control.
• Inactive repositories have their schedules disabled. After 60 days with no pushes, GitHub pauses scheduled workflows and emails the owner. Any commit re-enables them.
• A git push in a scheduled job can race. Two workflows pushing to the same branch produce a non-fast-forward rejection. The concurrency group prevents it; if you also push from elsewhere, git pull --rebase before pushing.
• Cron has no environment. No virtualenv, no PATH from your shell profile, no LANG — which is enough to turn a working script into a UnicodeEncodeError under cron only. Set the interpreter path absolutely and export what you need in the crontab.
• Minute budgets are real on private repositories. Public repositories run free on GitHub-hosted runners; private ones draw from a monthly allowance and then bill. A 10-minute hourly job is 7,200 minutes a month.
• Both are batch schedulers, not crawlers. Once the job is thousands of pages that must finish inside the interval, move the work to a queue — see Distributed Crawling with Celery and Redis — and keep the scheduler for kicking it off.
• Exit codes are your only signal on cron. Unlike Actions, nothing shows you a red cross. Pair cron with a heartbeat ping or a log-based check from Monitoring and Alerting for Scrapers.

Frequently Asked Questions

Cron or GitHub Actions, which should I pick? Use cron if you already run an always-on machine and need a precise start time, a stable IP or unlimited run minutes. Use GitHub Actions if you would rather not maintain a server at all, since it provides the runner, the Python install, encrypted secrets and log retention. For a small periodic scrape, Actions is usually the lower total effort.

Why did my scheduled workflow stop running? The most common cause is repository inactivity: GitHub disables scheduled workflows after 60 days without a push, and emails the owner. Any commit re-enables them. Otherwise check that the workflow file is on the default branch, that the cron expression is valid, and remember the schedule is evaluated in UTC.

How do I keep proxy passwords out of the workflow file? Store them as encrypted repository secrets and reference them as ${{ secrets.NAME }}, which exposes them to the step as masked environment variables. Read them with os.environ in your code, never interpolate them into a shell command that gets echoed, and never enable debug logging of request headers.

Can I commit scraped data back into the repository? Yes, and for small datasets it is genuinely useful — you get a free per-run history, a readable diff, and the repository activity keeps the schedule from being disabled. For large or frequent output, push to object storage instead, because every committed version is kept in the git history forever and cannot be pruned without a rewrite.

Related

• Deploying Scrapers to the Cloud — the parent topic comparing the places a scraper can live.
• Running Scrapers on AWS Lambda — the serverless alternative when a job needs to fan out rather than repeat.
• Monitoring and Alerting for Scrapers — knowing a scheduled run failed without reading logs by hand.
• Storing and Exporting Scraped Data — choosing where each run's output should land.