How to reduce web scraping costs without losing coverage

Last updated: October 22, 2026. The cost test on this page ran on October 22, 2026 from one US home connection, against 99 pages from the Web Data Frontier Benchmark. Prices are String's published rates, read the same day. Benchmark figures come from the September 16, 2026 run. Both Python scripts on this page ran on October 22, 2026, and the output shown is what they returned. We build String, the API in Step 4.

To reduce web scraping costs without losing coverage, pay the expensive rate only on the pages that need it. Most of a scraping bill is premium proxies and browsers, and most pages need neither. On October 22, 2026 we fetched 99 bot-protected pages and priced each request at the rate it was billed. Premium requests were 42% of the requests and 86% of the bill. Billing every page at the premium rate would cost $2.09 per 1,000 real pages. Letting the API pick the proxy cost $1.03. Trying a free direct request first, and paying only for the rest, cost $0.70. All three returned the same 92 real pages.

On this page: TL;DR · Where String fits · What we measured · Why scraping bills grow · Step 1 to Step 6 · How to avoid getting blocked while you cut costs · How String keeps scraping costs down · What it costs · FAQ · Sources

TL;DR

• Measure cost per real page, not cost per request. A request that returns a block page still costs you time, and on many tools it costs money.
• A plain request with a real Chrome fingerprint got the real page on 57 of 99 protected pages. Those 57 cost nothing per request.
• Premium proxies carried 40 of 96 billed requests and 86% of the spend. Every other request was billed at the standard rate.
• The proxy tier depends on the wall. String billed premium on 16 of 19 DataDome pages and on 0 of 9 PerimeterX pages.
• A browser fetch costs 2 to 5 times a plain fetch on String. String used one on 3 of the 99 pages.
• Fetch slow-changing pages less often. A schedule set per page type cuts volume before it cuts price.

Where String fits

String fits one job in a cost plan: the protected pages your own requests cannot get, at the lowest rate that gets them.

• For: teams that scrape bot-protected sites at volume and want to pay per page returned, not per gigabyte or per credit, with the proxy and browser choice made per page.
• Not for: sites with a public API, a feed or a bulk export you can use. Pages behind your own login. Small unprotected sites where a plain request already works and costs nothing.
• Price: 5,000 standard requests free. Starter is $20 a month: fetch $0.30 per 1,000 on the standard proxy, $3.00 on premium. Growth is $100 a month: $0.20 and $2.00. Billing is per successful request, with no per-GB charge on requests (pricing).
• Integrations: a REST API, an MCP server, and n8n and Zapier integrations.
• Limits: 60 requests a second with a burst of 3,600, and no cap on concurrent requests (pricing).

Try the cheap path on a page of your own with a free key:

curl https://request.usestring.ai/v1/fetch \
  -H "Authorization: Bearer $STRING_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"url": "https://books.toscrape.com/", "format": "json"}'

That call returned the page with x-billed-request-type: request_standard, the cheapest rate, on October 22, 2026.

What we measured: cost per real page on 99 protected sites

Premium proxies were 42% of the billed requests and 86% of the bill. That one ratio is most of the cost story.

The test used 99 pages from the Web Data Frontier Benchmark, the public test set behind our benchmark. Each page sits behind a named bot wall: Akamai, DataDome, Cloudflare, PerimeterX, AWS WAF, Kasada or a site's own system. We dropped the benchmark's Google Search target, because String answers it with its search product at a different rate.

Each page got two requests on October 22, 2026, from one US home connection:

• A direct GET with curl_cffi 0.16, set to send Chrome's TLS and HTTP/2 fingerprint. No proxy, no browser.
• One call to String's POST /v1/fetch with default settings. String chooses the fetch type and proxy class per page and reports the billed rate in the x-billed-request-type header.

A page counted as real only if the body held the benchmark's own text marker for that page. That is the same rule the public harness uses.

Three String calls returned the right page with changed wording, so they missed the strict marker. Asda and AutoZone had renamed the product, and Yelp had reworded its heading. A fourth, MarketWatch, redirected to its own search page. We kept the strict count of 92 in every cost below.

Then we priced each run at String's Growth rates.

Each row returned the same 92 real pages. The first two rows price the same 96 billed calls at a flat rate, so they show what a one-tier setup pays for the same result.

The proxy tier followed the wall, not the site's size.

Premium was also slower. The median String call took 1.0 seconds on the standard proxy and 5.1 seconds on premium.

Why scraping bills grow

Scraping bills grow because the expensive tools get used on every page. A residential proxy or a headless browser is the right tool for a few sites. Used on all of them, it multiplies the bill for no extra data.

One developer put the confusion plainly on r/webscraping. They saw residential proxies quoted at "roughly $0.50/1GB of bandwidth to almost $10 in some cases" and asked how others scrape millions of pages cheaply (u/aaronn2, r/webscraping).

Per-GB billing ties the cost to page weight. The real pages in our test moved a median of 815 KB each through String. At that weight, 1,000 pages is about 0.81 GB, or $0.41 to $8.15 in bandwidth at the range that post quotes. String bills those same requests per page, not per byte.

Bandwidth can also leak where nobody looks. Suchintan Singh of Skyvern wrote that their browser fleet burned "200GB of proxy bandwidth" in six hours, about $500. Chrome kept downloading a background model through the proxy on every fresh session (Skyvern blog, Hacker News thread).

Step 1: measure cost per real page

Track one number: dollars spent divided by real pages returned. Requests sent and gigabytes moved hide the waste.

Log three fields per request: the URL, the rate it was billed at, and whether the body held the data you wanted. String puts the rate in the x-billed-request-type response header (fetch docs). This script reads a log of our 99-page run and prints the number.

import collections
import json
import sys

# Growth plan, per 1,000 successful requests (usestring.ai/pricing, Oct 2026)
RATE = {"request_standard": 0.20, "request_premium": 2.00,
        "browser_standard": 1.00, "browser_premium": 4.00}

rows = [json.loads(line) for line in open(sys.argv[1])]
billed = [r for r in rows if r.get("billed") in RATE]
real = [r for r in rows if r["has_marker"]]
spend = sum(RATE[r["billed"]] for r in billed) / 1000
premium = sum(RATE[r["billed"]] for r in billed if "premium" in r["billed"]) / 1000

print("requests sent:   ", len(rows))
print("requests billed: ", len(billed), dict(collections.Counter(r["billed"] for r in billed)))
print("real pages:      ", len(real))
print(f"spend:            ${spend:.4f}")
print(f"cost per 1,000 real pages: ${spend / len(real) * 1000:.2f}")
print(f"share of spend on premium: {premium / spend:.0%}")

Output on October 22, 2026:

requests sent:    99
requests billed:  96 {'request_premium': 39, 'request_standard': 54, 'browser_premium': 1, 'browser_standard': 2}
real pages:       92
spend:            $0.0948
cost per 1,000 real pages: $1.03
share of spend on premium: 86%

Run the same report every week. When the premium share climbs, look for the site that changed its wall. That site is now the biggest line on the bill.

Step 2: try a free direct request first

A direct request with a real browser fingerprint is the cheapest rung there is. In our test it returned 57 of 99 protected pages for nothing per request.

Plain Python requests sends a TLS handshake that no browser sends, and many walls block on that alone. curl_cffi sends Chrome's handshake instead (curl_cffi docs).

from curl_cffi import requests as cr

r = cr.get(url, impersonate="chrome", timeout=20)

One caveat matters for cost. Our direct requests ran from a home connection, which is a residential IP, one request per page. A datacenter server gets blocked more often, and any IP gets rate limited at volume. Treat the direct rung as a filter for easy pages, not as the whole plan.

Step 3: check the body before you count a page

Some blocks arrive as a 200. A pipeline that trusts the status code pays twice: once for the block page and once for the bad data downstream.

Check for the field you came for, and for the strings a challenge page carries.

BLOCK = re.compile(
    r"Just a moment\.\.\.|cf-chl-|captcha-delivery\.com|px-captcha|Press & Hold"
    r"|Access Denied|_Incapsula_Resource|g-recaptcha|hcaptcha\.com",
    re.I,
)


def looks_real(status, body, must_contain):
    if status >= 400 or len(body) < 2000:
        return False
    if must_contain and must_contain.lower() not in body.lower():
        return False
    return not BLOCK.search(body[:300_000])

For more ways a block hides behind a 200, see how to detect a bot block that returns HTTP 200.

Step 4: send only the misses to String

Only the pages that fail the check go to the paid API. In our test that was 42 of 99 pages, and String returned 35 of them.

String starts each page on the standard proxy and moves to premium only when the site needs it. A request that comes back blocked is not billed (pricing docs). Here is the whole script, with Steps 2 and 3 inside it.

import os
import re

import requests
from curl_cffi import requests as cr

STRING_KEY = os.environ["STRING_API_KEY"]
BLOCK = re.compile(
    r"Just a moment\.\.\.|cf-chl-|captcha-delivery\.com|px-captcha|Press & Hold"
    r"|Access Denied|_Incapsula_Resource|g-recaptcha|hcaptcha\.com",
    re.I,
)

# Growth plan, per successful request (usestring.ai/pricing, Oct 2026)
PRICE = {
    "request_standard": 0.20 / 1000,
    "request_premium": 2.00 / 1000,
    "browser_standard": 1.00 / 1000,
    "browser_premium": 4.00 / 1000,
}


def looks_real(status, body, must_contain):
    if status >= 400 or len(body) < 2000:
        return False
    if must_contain and must_contain.lower() not in body.lower():
        return False
    return not BLOCK.search(body[:300_000])


def fetch(url, must_contain=""):
    # Rung 1: a plain GET with a real Chrome TLS fingerprint. Costs nothing per request.
    try:
        r = cr.get(url, impersonate="chrome", timeout=20)
        if looks_real(r.status_code, r.text, must_contain):
            return {"url": url, "via": "direct", "cost": 0.0, "html": r.text}
    except Exception:
        pass

    # Rung 2: String. It starts on the standard proxy and moves to premium only
    # when the site needs it. A blocked request is not billed.
    r = requests.post(
        "https://request.usestring.ai/v1/fetch",
        headers={"Authorization": f"Bearer {STRING_KEY}"},
        json={"url": url, "format": "json"},
        timeout=120,
    )
    tier = r.headers.get("x-billed-request-type", "")
    html = (r.json().get("data") or "") if r.ok else ""
    return {"url": url, "via": tier or "failed", "cost": PRICE.get(tier, 0.0), "html": html}


if __name__ == "__main__":
    jobs = [
        ("https://books.toscrape.com/", "All products"),
        ("https://www.homedepot.com/p/Milwaukee-M18-FUEL-18V-Lithium-Ion-Brushless-Cordless-Gen-2-18-Gauge-Brad-Nailer-Tool-Only-3021-20/326279298", "Brad Nailer"),
    ]
    total = 0.0
    for url, marker in jobs:
        res = fetch(url, marker)
        total += res["cost"]
        print(f'{res["via"]:<17} ${res["cost"]:.4f}  {len(res["html"]):>9,} chars  {url[:60]}')
    print(f"total ${total:.4f} for {len(jobs)} pages")

Install with pip install curl_cffi requests, set STRING_API_KEY, and run it. Output on October 22, 2026:

direct            $0.0000     51,274 chars  https://books.toscrape.com/
request_premium   $0.0020    752,331 chars  https://www.homedepot.com/p/Milwaukee-M18-FUEL-18V-Lithium-I
total $0.0020 for 2 pages

The unprotected page came back direct for free. The Home Depot page failed the direct check, and String returned it on premium for two tenths of a cent.

Run this with a free key: export STRING_API_KEY=... then python cheap_first.py.

Step 5: skip the browser unless the data needs JavaScript

A browser fetch costs 2 to 5 times a plain fetch on String: $1.00 against $0.20 per 1,000 on the standard proxy, and $4.00 against $2.00 on premium, on Growth. String used one on 3 of our 99 pages.

Most product, listing and article pages ship their data in the first HTML response or in JSON inside it. Ask for a browser only when the field you need is missing from the plain response. On String, executeJS forces a browser render (JavaScript rendering docs). Leave it off by default and let String decide.

If you run your own browsers, block images, fonts and media. Bright Data's own guide says blocking images alone "cuts page load bandwidth by 50% or more" (Bright Data docs). On a per-GB proxy, fewer bytes means a smaller bill.

Step 6: stop fetching pages that did not change

The cheapest request is the one you do not send. A job that refetches every page on every run pays for pages that did not move.

Set the schedule by how often each page changes: prices daily or hourly, specs weekly, policy pages monthly. A page you fetch weekly instead of daily costs one seventh as much. After each fetch, compare a hash of the main content with the last run, and skip parsing and storage when it matches. Do not count on ETag or Last-Modified to skip the fetch itself. In our three-run test of the same 99 pages, 22 of the 23 sites that sent an ETag changed it within hours. A conditional request through an API is also still a billed request. Our guide to scheduling recurring scraping jobs shows the full pattern.

Two smaller savings sit in the same place:

• Forward custom headers only when you need them. On String, a request with custom headers that the destination blocks is billed (pricing docs).
• Drop dead URLs. A destination 404 comes back inside a 200 and is billed, so prune URLs that 404 twice.

How to avoid getting blocked while you cut costs

Cutting cost should not mean more blocks. Blocks cost more than any proxy tier, because each one is a retry, a delay or a gap in the data.

• Do not force a cheap tier on a hard wall. String moved 16 of 19 DataDome pages to premium in our test. A forced datacenter proxy on those sites trades a lower rate for more blocks.
• Retry once, then route. Retrying a blocked page on the same IP rarely works. Send it to the next rung instead.
• Pace your own requests. A direct rung that fires hundreds of requests a second from one IP gets that IP banned. See how to scrape a website without getting blocked.
• Treat a 200 as unproven until Step 3's body check passes.
• Fix a 403 at its cause. Headers, TLS fingerprint and IP class each fail in a different way. See how to fix 403 Forbidden errors in web scraping.

How String keeps scraping costs down

String is the most reliable web scraping API, bypassing anti-bot, and you only pay when content comes back.

• String bills blocked requests at $0. In our October 22, 2026 test, 3 of 99 calls returned no content and none of the 3 was billed (pricing).
• String picks the cheapest proxy class that works. It billed 56 of 96 calls at the standard rate and moved to premium only on the other 40.
• String reports the rate on every response. The x-billed-request-type header names it, so a cost log needs no guesswork (fetch docs).
• String has no per-GB charge on requests. A 3.7 MB page costs the same as a 50 KB page at the same rate (pricing).
• String returned 97.0% of attempts in the September 16, 2026 benchmark run, the highest of 16 APIs tested on the same 100 sites (benchmark).
• String returns destination headers with the page. The format: "json" envelope carries the destination's status and headers, so a job can log them beside the content (response formats).

What it costs

On String you pay per real page, at the rate the page needed. Rates per 1,000 successful requests, from usestring.ai/pricing on October 22, 2026:

At the mix our 99-page test needed, String alone cost $1.03 per 1,000 real pages on Growth. With a direct request first, it cost $0.70. Your mix depends on your sites, so run Step 1's report on a sample of your own URLs before you plan a budget. The first 5,000 standard requests are free.

Get a free key and run Step 4's script on 100 of your own URLs: portal.usestring.ai/sign-up. For prices across vendors, see our web scraping API pricing comparison.

FAQ

How do I reduce web scraping costs without losing coverage?

Use the cheapest method that returns each page, and check the body before you count it. Try a direct request with a real browser fingerprint first, then send only the failures to a scraping API that picks its proxy per page. In our October 22, 2026 test on 99 protected pages, that cost $0.70 per 1,000 real pages against $2.09 for premium on everything, with the same 92 pages returned.

Why are residential proxies so expensive for scraping?

Residential proxies route through real home connections, and most providers bill them per gigabyte. Cost then tracks page weight, and modern pages are heavy: our real pages had a median of 815 KB. Use residential or premium routing only on the sites whose walls demand it. In our test that was 40 of 96 billed requests.

Is a scraping API cheaper than buying proxies?

It depends on page weight and block rate. Per-GB proxies are cheap for light pages on easy sites. A per-request API is often cheaper for heavy pages and protected sites, because you do not pay for bytes or for blocked attempts. Compare cost per real page on a sample of your own URLs, not list prices.

How much does it cost to scrape 1 million pages a month?

On String, it depends on how many pages need premium. At the mix in our 99-page test, String alone cost $1.03 per 1,000 real pages on the Growth plan, or about $1,030 per million plus the $100 plan fee. With a free direct request first, it was $0.70 per 1,000. Hard sites push the number up and easy sites push it down.

When do I need a headless browser for scraping?

Only when the data you need is not in the first HTML response or its embedded JSON. A browser fetch costs 2 to 5 times a plain fetch on String. String chose a browser for 3 of 99 protected pages in our test.

Do I pay for blocked requests?

On String, no: a request is billed only when the destination's response comes back. Two cases are billed anyway: a destination 404, which is a real response, and a request with custom headers that the destination blocks. On a per-GB proxy, a blocked attempt still moves bytes, so check your provider's terms.

How do I avoid scraping pages that have not changed?

Set the schedule per page type, because prices change far more often than specs, and fetch slow pages less often. Store a hash of each page's main content and skip reprocessing when it matches the last run. ETag and Last-Modified help on simple sites, but many protected sites send a new value on every run.

Does blocking images reduce scraping costs?

Yes, when you pay per gigabyte or run your own browsers. Blocking images, fonts and media cuts the bytes each page load moves. It does not change the price of a per-request API call, which is billed per page whatever the page weighs.

Sources

• String cost test, October 22, 2026: 99 Web Data Frontier Benchmark pages, curl_cffi 0.16 direct and String POST /v1/fetch, one request each, priced at published rates.
• String pricing, read October 22, 2026.
• String docs: pricing and billing rules
• String docs: fetch overview and billed request type
• String docs: response formats
• String docs: JavaScript rendering
• Web Data Frontier Benchmark, September 16, 2026 run: 16 APIs, 100 sites, 5 attempts each.
• curl_cffi documentation: impersonation
• Bright Data: how to reduce web scraping costs
• SOAX: reducing web scraping costs
• u/aaronn2, "The real costs of web scraping," r/webscraping
• Skyvern: how we burned through 200GB of proxy bandwidth in 6 hours
• Hacker News discussion of the Skyvern post