How to schedule recurring scraping jobs in Python

Last updated: October 22, 2026. The three scheduled runs on this page ran on October 22, 2026 from one US home connection, against 99 pages from the Web Data Frontier Benchmark. Benchmark figures come from the September 16, 2026 run. The Python job, the cron command and the APScheduler setup on this page ran on October 22, 2026, and the output shown is what they returned. We build String, the fetch API the job calls.

To schedule a recurring scraping job, write one run as a script that locks, fetches, checks each page and exits with an error when too much failed. Then let cron, GitHub Actions or a Python scheduler start it. The schedule is the easy part. A run that fails without telling you is the hard part.

On October 22, 2026 we ran one job three times, about 45 minutes apart, against 99 bot-protected pages through String's fetch API. 94 pages came back real on all three runs. One came back on some runs and not on others. Four never came back.

This page is the scheduling guide. To find the URLs a job should cover, see how to crawl an entire website. To cut what each run costs, see how to reduce web scraping costs.

On this page: TL;DR · Where String fits · What we measured · The named failure: a bad run that looks fine · Step 1 to Step 7 · How to avoid getting blocked on a schedule · How String keeps a scheduled job reliable · What it costs · FAQ · Sources

TL;DR

• Write the job as one run that ends. Let the scheduler start it. Do not write a loop that sleeps.
• Take a lock at the start. Two copies of the same job writing the same state is a common way a schedule goes wrong.
• Check each body for text that only the real page carries. MarketWatch returned 200 on all three runs with a search page instead of the stock page. A job that checks only the status stores it three times.
• Expect a few flips, and do not read them as changes. One page of 99 failed on run 1 and came back on runs 2 and 3. Four failed on every run, and an immediate retry fixed none of them. Those pages need a new URL or a different route.
• Send the page's ETag or Last-Modified back on the next run, but do not depend on it. 29 of 94 real pages sent one, and only 3 kept the same value across all three runs.
• Exit non-zero when too many pages fail, so cron mail, GitHub Actions or your alerting tells you.

Where String fits

String fits one job in a scheduled scraper: the fetch step, on the protected pages your own requests cannot get on every run.

• For: teams that run the same scrape every hour or every day on bot-protected sites (retail, travel, listings, news), and want one API call per page that comes back on run 1 and on run 300, with no proxy pool or browser fleet to keep alive between runs.
• Not for: sites with a feed, a sitemap with dates or a public API you can poll. Pages behind your own login. Unprotected sites where a plain request works on every run and costs nothing.
• Price: you pay only when content comes back. The first 5,000 standard requests are free. Starter is $20 a month plus $0.30 per 1,000 standard fetches and $3.00 with premium proxies. Growth is $100 a month at $0.20 and $2.00 (pricing).
• Integrations: a REST API you call from any scheduler, an MCP server for agents, and String Actors on Apify Store.
• Limits: You bring the scheduler (cron, GitHub Actions, APScheduler or an orchestrator), and String handles the fetch. The rate limit is 60 requests a second per organization, with no cap on concurrent requests (pricing, rate limits). String does not store page bodies between runs: on a 304, you reuse your own copy (custom headers).

What we measured: one job, three runs, 99 protected pages

We ran one job three times on October 22, 2026: at 14:21, 15:12 and 15:58 UTC. Each run fetched the 100 target pages of the Web Data Frontier Benchmark through String's /v1/fetch with "format": "json", one call per page, 16 at a time, with no retry.

The Google results URL is answered by String's search endpoint, not fetched as a page, so we left it out: 99 pages. A page counts as real when the destination status is 2xx and the body holds the benchmark's own text from that page, such as the product name. That is the benchmark's rule, so a challenge page served at 200 counts as a failure.

Across the three runs:

The one flip was Mouser, an Akamai site. String answered 502 on run 1, then returned the product page on runs 2 and 3. A job that deleted or alerted on missing pages would have fired once for nothing.

The four that never came back split two ways. Two are dead URLs: a CarGurus listing that answers 404, and a MarketWatch stock page that now redirects to a search page with a 200. Two are hard sites: eMAG failed at the destination on all three runs, and Temu timed out or failed on all three. An immediate retry after run 3 got none of the four.

Four pages count by hand on every run. The site had changed the words our check looks for, and the page title showed the right page: an ASDA and an AutoZone product renamed, a Yelp heading reworded, and a New York Times live page that had moved on to newer coverage. Without them the counts drop by four.

Caveats: three runs in under two hours is a short window. A job that runs for weeks will meet site changes, new walls and dead URLs that we did not. One home connection called the API; the pages were fetched by String.

The named failure: a bad run that looks fine

The named failure is a run that finishes, exits 0 and stores the wrong thing. It takes four forms on a schedule:

1. Overlapping runs. A run takes longer than the interval, the next one starts, and both write the same state. DigitalOcean community answerer Bobby Iliev puts it this way: "if the scraper takes longer than an hour you will end up with overlapping runs" (DigitalOcean Community, June 2026). Step 1 fixes it.
2. A 200 that is not the page. MarketWatch answered 200 on all three of our runs. The body was a search page, because the stock page now redirects there. A job that checks only the status code stores that search page as data on every run. Step 2 fixes it.
3. Flips read as changes. A page that failed on one run and came back on the next looks like a page that vanished and reappeared. If your job deletes or alerts on it, you get noise. Step 4 keeps the last good copy.
4. Unchanged pages reprocessed. Every run refetches and reparses pages that did not change. Step 3 sends validators back so the site can answer 304 Not Modified.

The scheduler can fail too. In late August 2026, developers on GitHub's community forum reported scheduled workflows running hours late, or not at all:

"I have three jobs scheduled to run every two hours. Overnight I got a single run of each job at 00:26, 00:36 and 06:16 UTC and that was it! It didn't run once since." steeviebops, GitHub Community, August 31, 2026

GitHub's own docs say scheduled runs can be delayed under load and that "some queued jobs may be dropped" (GitHub Docs). Step 6 covers what to do about it.

Step 1: write one run that locks

Write the job as a script that does one pass and exits. The scheduler decides when it runs. The script decides whether it may run: it takes a file lock first, and exits quietly if another copy holds it.

The full script is below. We ran it on Python 3.12 with requests 2.34.2. It needs a String API key in STRING_API_KEY. Save it as scrape_job.py.

"""One run of a recurring scraping job. Schedule it with cron, GitHub Actions or APScheduler.

What one run does:
  1. takes a lock, so two runs never overlap
  2. fetches each URL through String's fetch API, sending the last ETag / Last-Modified back
  3. checks the body for text only the real page carries, so a block page never counts as data
  4. hashes the visible text and compares it with the last good run
  5. saves state only for good pages, and exits non-zero when too many pages failed
"""
import fcntl, hashlib, html, json, os, random, re, sys, time
from pathlib import Path

import requests

API = "https://request.usestring.ai/v1/fetch"
KEY = os.environ["STRING_API_KEY"]
HERE = Path(__file__).resolve().parent
URLS = json.loads((HERE / "urls.json").read_text())
STATE_FILE = HERE / "state.json"
MAX_FAILED_SHARE = 0.2  # exit 1 if more than 20% of pages fail, so the scheduler reports it


def visible_text(body: str) -> str:
    body = re.sub(r"<(script|style|noscript|svg|template)\b.*?</\1\s*>", " ", body, flags=re.S | re.I)
    text = html.unescape(re.sub(r"<[^>]+>", " ", body))
    text = re.sub(r"\b[0-9a-f]{16,}\b", " ", text, flags=re.I)  # session ids, cache busters
    return re.sub(r"\s+", " ", text).strip()


def fetch(url: str, saved: dict) -> dict:
    payload = {"url": url, "format": "json"}
    # Custom headers force a plain (non-browser) fetch, so skip validators on pages that needed a browser.
    if not saved.get("method", "").startswith("browser"):
        conditional = {}
        if saved.get("etag"):
            conditional["If-None-Match"] = saved["etag"]
        if saved.get("last_modified"):
            conditional["If-Modified-Since"] = saved["last_modified"]
        if conditional:
            payload["headers"] = conditional
    for attempt in range(4):
        r = requests.post(API, json=payload, timeout=120,
                          headers={"Authorization": f"Bearer {KEY}"})
        if r.status_code == 429 or r.status_code >= 500:
            retry_after = r.headers.get("Retry-After", "")
            wait = int(retry_after) if retry_after.isdigit() else 2 ** attempt * 5 + random.uniform(0, 3)
            time.sleep(wait)
            continue
        r.raise_for_status()
        return r.json(), r.headers.get("x-billed-request-type", "")
    raise RuntimeError(f"gave up after 4 tries, last status {r.status_code}")


def run() -> int:
    lock = open(HERE / ".scrape.lock", "w")
    try:
        fcntl.flock(lock, fcntl.LOCK_EX | fcntl.LOCK_NB)
    except BlockingIOError:
        print("previous run still going, skipping this one")
        return 0

    state = json.loads(STATE_FILE.read_text()) if STATE_FILE.exists() else {}
    failed = 0
    for item in URLS:
        url, must_contain = item["url"], item["mustContain"]
        saved = state.get(url, {})
        try:
            env, method = fetch(url, saved)
        except Exception as e:
            print(f"FAILED     {url}  {e}")
            failed += 1
            continue
        status = env.get("statusCode")
        headers = {k.lower(): v if isinstance(v, str) else v[0] for k, v in (env.get("headers") or {}).items()}
        if status == 304:
            print(f"UNCHANGED  {url}  304, reuse the saved copy")
            continue
        body = env["data"] if isinstance(env.get("data"), str) else json.dumps(env.get("data"))
        if not (status and 200 <= status < 300) or must_contain not in body:
            print(f"BLOCKED    {url}  status {status}, page text missing; state kept")
            failed += 1
            continue
        text = visible_text(body)
        if item.get("watch"):  # compare only the field you act on, such as the price
            m = re.search(item["watch"], text)
            if not m:
                print(f"NO FIELD   {url}  watch pattern not found; state kept")
                failed += 1
                continue
            text = m.group(0)
        digest = hashlib.sha256(text.encode()).hexdigest()
        verdict = "UNCHANGED" if digest == saved.get("hash") else ("NEW" if not saved else "CHANGED")
        print(f"{verdict:10} {url}  {len(body):,} bytes")
        state[url] = {
            "hash": digest,
            "etag": headers.get("etag"),
            "last_modified": headers.get("last-modified"),
            "method": method,  # how String fetched it: request_standard, request_premium, browser_...
            "fetched_at": time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime()),
        }
        # write the page wherever your pipeline reads it; a CHANGED page is the one to process

    STATE_FILE.write_text(json.dumps(state, indent=2))
    print(f"done: {len(URLS) - failed} ok, {failed} failed")
    return 1 if failed > MAX_FAILED_SHARE * len(URLS) else 0


if __name__ == "__main__":
    sys.exit(run())

fcntl.flock with LOCK_NB fails at once when another process holds the lock, so the second run prints a line and exits 0. The lock is released when the process ends, even after a crash. fcntl works on Linux and macOS. On Linux you can also put the lock in the cron line with flock -n (flock man page).

Put the pages in urls.json next to the script. Give each URL a phrase that only the real page contains. Add watch when you care about one field, such as a price:

[
  {"url": "https://arxiv.org/abs/1706.03762", "mustContain": "Attention Is All You Need"},
  {"url": "https://www.allmenus.com/il/chicago/223447-chicagoland-pizza-pasta/menu/", "mustContain": "Chicagoland Pizza"},
  {"url": "https://www.homedepot.com/p/Milwaukee-M18-FUEL-18-Volt-Lithium-Ion-Brushless-Cordless-Gen-II-18-Gauge-Brad-Nailer-Tool-Only-2746-20/309752194",
   "mustContain": "Brad Nailer", "watch": "\\$ ?[0-9,]+ ?\\. ?[0-9]{2}"}
]

Run it twice by hand, then twice at the same moment. On October 22, 2026 it printed:

== run 1 2026-10-22T16:08:01Z
NEW        https://arxiv.org/abs/1706.03762  44,230 bytes
NEW        https://www.allmenus.com/il/chicago/223447-chicagoland-pizza-pasta/menu/  126,184 bytes
NEW        https://www.homedepot.com/p/Milwaukee-M18-FUEL-18-Volt-Lithium-Ion-Brushless-Cordless-Gen-II-18-Gauge-Brad-Nailer-Tool-Only-2746-20/309752194  785,176 bytes
done: 3 ok, 0 failed
exit 0
== run 2 2026-10-22T16:08:02Z
UNCHANGED  https://arxiv.org/abs/1706.03762  304, reuse the saved copy
UNCHANGED  https://www.allmenus.com/il/chicago/223447-chicagoland-pizza-pasta/menu/  304, reuse the saved copy
UNCHANGED  https://www.homedepot.com/p/Milwaukee-M18-FUEL-18-Volt-Lithium-Ion-Brushless-Cordless-Gen-II-18-Gauge-Brad-Nailer-Tool-Only-2746-20/309752194  785,176 bytes
done: 3 ok, 0 failed
exit 0
== runs 3 and 4 started together 2026-10-22T16:08:03Z
previous run still going, skipping this one
exit 0
UNCHANGED  https://arxiv.org/abs/1706.03762  304, reuse the saved copy
UNCHANGED  https://www.allmenus.com/il/chicago/223447-chicagoland-pizza-pasta/menu/  304, reuse the saved copy
UNCHANGED  https://www.homedepot.com/p/Milwaukee-M18-FUEL-18-Volt-Lithium-Ion-Brushless-Cordless-Gen-II-18-Gauge-Brad-Nailer-Tool-Only-2746-20/309752194  785,177 bytes
done: 3 ok, 0 failed
exit 0

The second run got 304 from arXiv and Allmenus and found the Home Depot price unchanged. The fourth run started while the third held the lock, and skipped.

Step 2: check the page body

A 200 does not mean you got the page. The job marks a page as blocked when the destination status is not 2xx or the body does not hold the mustContain phrase. A blocked page counts as a failure and does not touch the saved state.

In our three runs, 13 of 297 page fetches failed. Three of the 13 came back with a 200: the MarketWatch search page, once per run. The other ten were a 404 or an error, which any status check catches. For the patterns a block page takes, see how to detect a bot block that returns HTTP 200.

Run this with a free key to see what your own pages return:

curl https://request.usestring.ai/v1/fetch \
  -H "Authorization: Bearer $STRING_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"url": "https://arxiv.org/abs/1706.03762", "format": "json"}'

On October 22, 2026 it returned statusCode 200, the arXiv page as 44,230 characters of HTML in data, and the site's own Last-Modified and Cache-Control in headers. Header values came back as strings, with the site's own capitalization, so the script lowercases the names before it reads them.

Step 3: send validators back on the next run

Save the page's ETag and Last-Modified with each good page. On the next run, send them back as If-None-Match and If-Modified-Since in String's headers field. When the page has not changed, the site answers 304, String returns it inside an HTTP 200 with "statusCode": 304 and an empty data, and you reuse your saved copy (custom headers). HTTP defines both headers in RFC 9110.

Do not count on it. On run 1, 29 of the 94 real pages carried an ETag, a Last-Modified date or both. Only 3 kept the same value across all three runs: arXiv, Allmenus and Crunchbase. 22 of the 23 ETags changed between runs, and 5 Last-Modified dates were stamped within 15 minutes of the fetch.

Minutes after run 3, we sent each of the 29 pages its run 3 validators back through String. 7 answered 304. The other 22 answered 200 with a body.

So treat a 304 as a bonus. The text hash is the check that works on every page. Hash only the part you act on.

A whole-page hash moves too often: by our hash of the visible text, 68 of the 94 pages that came back on every run differed on at least one run. For example, the Home Depot product page changed between two fetches a few seconds apart, because the review count went from 2,926 to 2,927, while the price stayed $329.00. A watch pattern on the price stops that from counting as a change.

Two rules come with the headers field. Custom headers are sent on plain fetches only, never in a browser, so skip validators on pages that needed a browser.

Skyscanner is one: String fetched it in a browser on run 1, and the conditional request, sent without a browser, came back as a 708-byte page with none of the page text. The script stores the x-billed-request-type of each good fetch and leaves the headers off when it starts with browser (custom headers). And a request with custom headers that the site blocks is billed, unlike other failed requests (billing).

A 304 through String saves the site's bandwidth and your parse time. It is billed like any other successful fetch (custom headers).

Step 4: retry the page, keep the last good copy

A page that fails on one run often comes back. In our test, one page flipped: Mouser failed with a 502 on run 1 and came back on runs 2 and 3. The four pages that failed on run 3 had failed on every run, and an immediate retry got none of them. A retry helps a flip. It does not help a dead URL.

The script does three things about it:

• It retries an API 429 or 5xx up to four times, waiting for Retry-After when the response sends one and backing off with jitter when it does not (rate limits). For more on backoff, see how to handle rate limiting when scraping.
• It keeps the last good copy of a failed page. A failed page is never written to state, so a flip does not look like a change.
• It exits 1 when more than 20% of pages fail. GitHub Actions marks that run failed. Cron ignores exit codes, but it mails any output a job prints to the crontab's owner, or to MAILTO when you set it (crontab). Step 5 turns the exit code into one line of output.

Watch a page that fails run after run. That is a moved URL, a dead listing or a new wall. In our runs CarGurus answered 404 for a listing that is gone, and MarketWatch redirected to a search page.

Step 5: schedule it with cron

Cron is the simplest scheduler on a Linux server or a Mac that stays on. Put the key in a .env file next to the script, readable only by you:

export STRING_API_KEY="your key"

Then run crontab -e and add one line. This runs the job every hour at minute 17:

17 * * * * cd /home/you/scraper && . ./.env && .venv/bin/python scrape_job.py >> scrape.log 2>&1 || echo "scrape job failed, see scrape.log"

Cron starts jobs with a minimal environment, so use full paths and load the key in the line itself (crontab).

The job's own output goes to scrape.log. Only the echo reaches cron's mail, and only when the job exits non-zero. We ran that command under env -i /bin/sh, a bare environment like cron's, on October 22, 2026. It printed nothing and wrote done: 3 ok, 0 failed to scrape.log. With a mustContain phrase the page does not hold, it wrote BLOCKED to the log and printed scrape job failed, see scrape.log.

Pick a minute that is not 0. GitHub's docs name the start of every hour as its busiest time for scheduled runs, and many other jobs start on the hour too.

Step 6: or schedule it with GitHub Actions

GitHub Actions runs the job without a server. Add the key as a repository secret named STRING_API_KEY, then save this as .github/workflows/scrape.yml:

name: scrape
on:
  schedule:
    - cron: "17 * * * *"   # every hour at :17 UTC; avoid :00, the busiest minute
  workflow_dispatch:        # a Run workflow button for tests and missed runs
concurrency:
  group: scrape
  cancel-in-progress: false # let a slow run finish; the next one waits
jobs:
  scrape:
    runs-on: ubuntu-latest
    timeout-minutes: 30
    steps:
      - uses: actions/checkout@v7
      - uses: actions/setup-python@v7
        with:
          python-version: "3.12"
      - run: pip install requests==2.34.2
      - uses: actions/cache/restore@v6
        with:
          path: state.json
          key: scrape-state-${{ github.run_id }}
          restore-keys: scrape-state-
      - run: python scrape_job.py
        env:
          STRING_API_KEY: ${{ secrets.STRING_API_KEY }}
      - uses: actions/cache/save@v6
        if: always()
        with:
          path: state.json
          key: scrape-state-${{ github.run_id }}

What each part does:

• schedule uses POSIX cron syntax in UTC. You can add timezone: with an IANA name. The shortest interval is every 5 minutes (GitHub Docs).
• concurrency with cancel-in-progress: false lets a slow run finish, and holds at most one waiting run (concurrency docs). It does the job the file lock does on a server.
• actions/cache carries state.json from one run to the next, because each run starts on a clean machine (dependency caching).
• workflow_dispatch adds a Run workflow button, which you need when a scheduled run is dropped.

Three limits to plan for. Scheduled runs can start late or be dropped under load, most often at the start of the hour.

In a public repository, scheduled workflows turn off after 60 days with no repository activity (GitHub Docs). And GitHub removes caches that nobody reads for 7 days, so a weekly job loses its state (dependency caching). If the schedule must hold to the minute, trigger the workflow from an outside scheduler through workflow_dispatch, as developers in that GitHub thread ended up doing.

We checked this workflow with actionlint 1.7.12 on October 22, 2026: no findings. We did not run it on GitHub. Run it once with the Run workflow button before you trust the schedule.

Step 7: or schedule it inside Python with APScheduler

When the job lives inside a long-running Python service, use APScheduler (3.11.3 on PyPI, October 2026). Save this next to scrape_job.py as run_forever.py:

"""Keep one Python process running and let it call the job on a schedule."""
from apscheduler.schedulers.blocking import BlockingScheduler

from scrape_job import run

scheduler = BlockingScheduler(timezone="UTC")
scheduler.add_job(
    run,
    "cron",
    minute=17,              # every hour at :17
    max_instances=1,        # never two runs at once
    coalesce=True,          # after downtime, run once, not once per missed slot
    misfire_grace_time=600, # still run if the process wakes up to 10 minutes late
)
scheduler.start()

The three options are the ones that matter for scraping (APScheduler user guide):

• max_instances=1: by default APScheduler allows only one instance of each job at a time, and a start that finds the last run still going is skipped as a misfire. Setting it says so in the code.
• coalesce=True: after downtime, the missed starts roll into one run instead of running back to back.
• misfire_grace_time=600: a start that comes up to 10 minutes late still runs.

On October 22, 2026 at 16:09 UTC, run_forever.py registered cron[minute='17'] with its next run at 16:17 UTC. A test copy with a 20-second interval ran the job three times in 65 seconds, each time with done: 3 ok, 0 failed.

The process has to stay up. Run it under systemd, a container restart policy or a process manager, or use cron and let each run start fresh.

Which scheduler to use

Not everyone wants cron. A developer with hundreds of Python scrapers asked r/webscraping for a scheduler that can scale, and added: "I want something lightweight I don't want to do cron" (u/startup_biz_36, r/webscraping, November 2024). The job above runs the same way under each of these:

How to avoid getting blocked on a schedule

A schedule makes your traffic regular, and regular traffic is easy to spot. The checklist, in the order that pays off:

• Use a feed, a sitemap with lastmod dates or a public API when the site has one.
• Run no more often than the data changes. Hourly prices and daily listings are common; most pages do not need more.
• Pick an odd minute and add a little jitter before each run.
• Pace requests per site. Do not open 50 connections to one host because the job has 50 URLs.
• Send validators back (Step 3), so an unchanged page can come back as a small 304.
• Check every body for page text (Step 2), and keep the last good copy when a page fails (Step 4).
• Send the pages that your own requests cannot get to an API built for blocked sites. For the techniques behind each step, see how to scrape without getting blocked. For a no-code version of the same job, see how to scrape websites in n8n without getting blocked.

How String keeps a scheduled job reliable

String is the most reliable web scraping API, bypassing anti-bot, and you only pay when content comes back.

• String returned 94 of 99 bot-protected pages on all three runs of one job on October 22, 2026, about 45 minutes apart. Run by run it returned 94, 95 and 95. The one page that flipped, Mouser, failed once and came back on the next run.
• String had the highest success rate of 16 scraping APIs in the September 16, 2026 benchmark run: 97.0% across 100 sites, five attempts per site (benchmark).
• "format": "json" returns the site's own status code and response headers with every page. A job reads statusCode, etag and last-modified from one response (fetch reference).
• String forwards If-None-Match and If-Modified-Since and passes a 304 through. Minutes after the third run, 7 of the 29 pages that carried a validator answered 304 through String with an empty body (custom headers).
• A failed request is not billed. A job that hits a wall on one run pays nothing for that page, unless the request carried custom headers (billing).
• String picks a plain request, a premium proxy or a browser per page, so the job has one code path. The x-billed-request-type response header names the method each call used (API overview).

What it costs

A job's cost is pages times runs. 1,000 pages a day is about 30,000 fetches a month. On Starter that is $9 in usage at the standard rate, plus the $20 plan, and more for pages that need premium proxies or a browser (pricing).

A failed request is not billed. A destination 404 that comes back is billed, and so is a blocked request that carried custom headers (billing). A 304 is billed like any other fetch, so validators save the site's bandwidth and your processing, not String credits.

The cheapest call is the one you do not make: run only as often as the data changes, and drop URLs that answer 404 run after run. In our runs the dead CarGurus listing came back as a billed 404 on all three. For the full cost plan, including which pages need premium proxies, see how to reduce web scraping costs and the web scraping API pricing comparison.

Start with the job above, a free key and your own urls.json. The first run prints how many of your pages come back.

FAQ

How do I schedule and run recurring scraping jobs?

Write the scraper as a script that does one pass and exits, with a lock, a body check and an exit code. Then start it with cron, GitHub Actions or APScheduler. In our test of one job run three times on 99 protected pages, 94 came back on every run and one flipped, so plan for a failed page on any run.

How can I run a web scraper automatically every hour?

On a server, add 17 * * * * and the full command to your crontab. Without a server, use a GitHub Actions workflow with schedule: - cron: "17 * * * *". Inside a Python service, use APScheduler with minute=17. Pick a minute other than 0.

How do I stop cron jobs from overlapping?

Take a lock at the start of the run. In Python, fcntl.flock with LOCK_NB makes a second copy exit at once. On Linux, flock -n in the cron line does the same. In GitHub Actions, use a concurrency group.

How can I automate web scraping without a server?

Use a GitHub Actions schedule trigger and keep state in actions/cache, or a cloud scheduler that starts a container or function. GitHub can start scheduled runs late or drop them under load, so add a manual trigger and an outside check.

How do I know if a scheduled scrape failed?

Check each page body for text only the real page carries, count the failures, and exit non-zero above a threshold. In our three runs, MarketWatch returned 200 every time with a search page, so a status check alone would have missed it.

Should I use ETag or Last-Modified in a recurring scraper?

Yes, as a bonus. Send them back as If-None-Match and If-Modified-Since and reuse your copy on a 304. Only 3 of 99 protected pages kept the same validator across our three runs, so a content hash is still needed.

How often should a scraping job run?

As often as the data changes and no more: hourly for prices on fast-moving marketplaces, daily for most catalogs and listings, weekly for slow pages. Each extra run adds load on the site and cost for you.

What is the best Python scheduler for web scraping?

Cron for one server, GitHub Actions when you have no server, and APScheduler when the job already lives in a Python process. Move to Airflow, Prefect or Dagster when you have many jobs that depend on each other.

Sources

• Scheduled-job test, October 22, 2026: three runs (14:21, 15:12 and 15:58 UTC) of one job over the 100 Web Data Frontier Benchmark target pages through String /v1/fetch (format: json), one call per page per run, no retry; an immediate retry of the 4 pages that failed on the third run; conditional requests on 29 pages with the validators from the third run.
• Web Data Frontier Benchmark, September 16, 2026 run: 16 APIs, 100 sites, five attempts per site. Harness.
• GitHub Docs: schedule event, concurrency, dependency caching
• GitHub Community discussion #156282: delays in scheduled workflows
• DigitalOcean Community: schedule a Python web scraper every hour
• Reddit r/webscraping: What tool are you using for scheduling web scraping tasks?
• APScheduler 3.x user guide, APScheduler 3.11.3 on PyPI
• flock(1), crontab(5)
• RFC 9110: conditional requests
• String fetch reference, custom headers and conditional revalidation, API overview, rate limits, MCP server
• String pricing, read October 22, 2026