NewLaunching String Web Access APIRead the manifesto →
String
Book demoSign up
← Answers

Which scraping API survives bot protection upgrades?

String team · Updated October 28, 2026

The scraping API that keeps working through bot protection upgrades is the one you can show held up on the same sites, run after run. We ran 15 APIs against the same 90 sites on July 15, August 11 and September 16, 2026. On 25 sites, bot protection got harder between July and September. String returned 23 of those 25 sites at 4 of 5 attempts or better in all three runs. Firecrawl and Scrapfly held 13 each.

String builds this benchmark and ranks first in it. The harness and every run file are public, so you can rerun each number on this page. Firecrawl and Scrapfly ran some or all of these runs below the anti-bot mode they offer today, so their numbers may understate them. That caveat, and the others, sit next to each vendor's numbers below.

How String keeps returning protected pages

String is the most reliable web scraping API, bypassing anti-bot, and you only pay when content comes back. Here is what that means when a site upgrades its protection, one fact per line:

  • String held 23 of 25 sites whose protection got harder. It returned each at 4 of 5 attempts or better in the July 15, August 11 and September 16, 2026 runs, and 117 of 125 attempts on those sites in September.
  • String was stable on 84 of 90 sites across all three runs. The next highest was Scrapfly at 61.
  • String returned 97.0% of 500 requests across 100 sites in the September 16, 2026 run, the highest of 16 APIs (benchmark).
  • One call picks the route each page needs. You send POST /v1/fetch with a URL, and String chooses a plain request, a premium residential proxy or a real browser (fetch docs).
  • The response names the route it took. On October 14, 2026, the x-billed-request-type header read request_premium for G2 and Yelp, so you can see when a site starts needing the heavier path.
  • A blocked request costs nothing. String bills only when content comes back, so a site that hardens overnight does not run up your bill while it fails.

Where String fits

String is for teams that run scheduled scrapes of protected public pages, such as prices, listings, reviews or job posts, and need the job to come back the morning after a site changes its defenses.

  • For: recurring jobs on bot-protected sites, AI agents that fetch live pages, and teams that would rather not maintain a stealth browser fleet.
  • Not for: pages behind a login, form submission or checkout flows, or a site whose official API already gives you the data.
  • Price: on the Starter plan ($19 a month), a plain fetch is $0.25 per 1,000 on standard proxies and $2.40 on premium; a browser fetch is $1.20 and $4.80 (pricing). Protected sites often take the premium path, so budget at that rate. A blocked request is not billed.
  • Integrations: a REST API from any language, a hosted MCP server for agents, n8n and Zapier integrations, and LangChain and AI SDK guides.
  • Limits: 15 requests a second on Starter and 30 on Growth, with up to 10,000 concurrent fetches. String fetches public pages only.

What three benchmark runs show

25 of 90 sites got harder between July and September 2026. We call a site harder when four reference APIs, on average, fell 20 points or more from the July run to the September run. Ten sites moved the other way.

The four reference APIs are Bright Data, Browserbase, Context.dev and Oxylabs. We picked them because their harness setup did not change across the three runs and none of them hit an outage of their own API. A drop on their side is the best signal the run data gives that the site itself changed.

This is a proxy. No site told us it upgraded its protection, and a page can get harder for other reasons, such as an expired listing or a change in how the site serves that URL. Treat "harder" as "four unchanged APIs found it much harder", nothing more.

Some of the 25 sites, with the reference group's average success in July, August and September:

Site Reference group, Jul / Aug / Sep String, of 5 attempts, Jul / Aug / Sep
G2 55% / 70% / 0% 5 / 5 / 5
Allegro 50% / 55% / 0% 5 / 4 / 4
Yelp 50% / 35% / 5% 5 / 5 / 4
CarGurus 100% / 100% / 25% 5 / 5 / 5
Coupang 75% / 10% / 25% 5 / 4 / 4
Etsy 65% / 75% / 30% 5 / 5 / 5
Booking.com 100% / 50% / 50% 5 / 4 / 5

The pattern on G2 is the one this page is about. Three of the four reference APIs returned the G2 category page in July. In September none of them did. String returned it on 5 of 5 attempts in all three runs.

To test one of these pages yourself, send it to String with a free key:

curl -X POST https://request.usestring.ai/v1/fetch \
  -H "Authorization: Bearer $STRING_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"url":"https://www.g2.com/categories/emerging-ai-software"}'

The first 20,000 standard fetches are free.

Every API on the 25 harder sites

15 APIs ran in all three runs. "Held" means 4 of 5 attempts or better on that site in July, August and September. "Stable on 90" is the same test across every site in all three runs, which is the fairer view for the four reference APIs (see the note under the table).

API Held, of 25 harder sites September successes on those sites Stable on all 90 sites Fell on, of 90
String 23 117 of 125 84 1
Firecrawl* 13 106 of 125 54 2
Scrapfly* 13 106 of 125 61 3
ScraperAPI* 12 101 of 125 46 4
Bright Data 10 64 of 125 48 10
Nimble* 9 62 of 125 42 7
ZenRows 8 of 18 65 of 90 23 of 50 4
Zyte* 8 64 of 125 45 6
Oxylabs 8 59 of 125 45 7
Context.dev 8 44 of 125 54 12
Decodo 5 of 22 59 of 110 27 of 71 5
ScrapingAnt* 3 25 of 125 19 4
Browserbase 3 20 of 125 30 8
ScrapingBee* 2 70 of 125 20 of 88 3
Scrapingdog* 2 47 of 125 17 3

"Fell on" counts sites where the API returned 4 or 5 of 5 in July and 0 or 1 of 5 in September.

Read the table with these four notes:

  • The reference APIs look worse here by design. Bright Data, Browserbase, Context.dev and Oxylabs define which sites count as harder, so their own scores on those 25 sites drop by construction. Compare them on the 90-site columns.
  • Eight APIs carry an asterisk because their setup changed or sits below their best mode. Firecrawl, Scrapfly, ScraperAPI, Zyte, ScrapingAnt, ScrapingBee, Scrapingdog and Nimble ran some or all of these runs on a setting below the anti-bot mode they offer today. The September 16 run moved ScrapingBee, ScraperAPI, Scrapfly, Scrapingdog and Nimble to the settings their vendors document for bot-protected pages. A run-over-run gain for these eight can come from that setup change, not the product.
  • Three APIs had outages of their own, and those attempts are left out. ZenRows' API returned HTTP 402 in August and closed connections in September; Decodo's returned HTTP 429 in September; ScrapingBee's returned HTTP 401 for the last 11 sites in August. Those attempts measured the vendor's API, not the site, so they are not scored. That is why some rows count fewer sites.
  • Apify ran only in September, so it has no three-run record and is not in the table.

Why bot protection upgrades break scrapers

Cloudflare and DataDome change what they check, and a bypass tuned to the old checks can stop passing overnight. The developer behind a favicon service saw it happen on sites they did not run:

"Lately, more and more of my scraper's requests hit Cloudflare bot protection - and since no icons are present on that page, the result is blank, and my API serves a fallback image."

user77116, Cloudflare Community, March 2023

Two details in that post matter. The failure was silent: the service kept answering, with a blank result. And the accepted answer from a community MVP was to apply to Cloudflare's verified bots program, which is the right first step if you run a well-behaved crawler that sites want to let in.

The drop can also be total. A developer using DataDome CAPTCHA solvers wrote:

"It was working with a 100% success rate on a hundred requests, now it is 0%."

u/Due-Exercise6990, r/webscraping, January 2025

The author guessed DataDome had changed something and the solvers would need time to catch up. That wait is the cost a scraping API is supposed to absorb for you. The benchmark measures whether it does.

If your own job just went quiet, check the content before you change tools. A 200 with a challenge page is the most common silent failure; our page on detecting a bot block behind HTTP 200 shows the checks. Then run the failing URLs through the script in the next section with a free key and compare.

Python: check your hardest URLs on a schedule

This script sends each URL to String's fetch endpoint and judges the result by the page's own text, not by the status code. Run it daily on the URLs your job depends on, and a protection change shows up as real_page: False before it shows up as missing data.

import os
import re
import sys

import requests

API = "https://request.usestring.ai/v1/fetch"
HEADERS = {
    "Authorization": f"Bearer {os.environ['STRING_API_KEY']}",
    "Content-Type": "application/json",
}


def check(url: str, must_contain: str) -> dict:
    r = requests.post(API, headers=HEADERS, json={"url": url, "format": "json"}, timeout=120)
    r.raise_for_status()
    body = r.json()
    html = str(body.get("data", ""))
    title = re.search(r"<title[^>]*>(.*?)</title>", html, re.S | re.I)
    return {
        "url": url,
        "status": body.get("statusCode"),  # the target site's own status code
        "real_page": must_contain.lower() in html.lower(),  # judge by content, not status
        "title": title.group(1).strip()[:60] if title else None,
        "bytes": len(html),
        "route": r.headers.get("x-billed-request-type"),  # standard or premium path
    }


if __name__ == "__main__":
    # usage: python check_hardened.py URL "text the real page contains" [URL "text" ...]
    args = sys.argv[1:]
    for url, text in zip(args[::2], args[1::2]):
        print(check(url, text))

We ran it on October 14, 2026 with pip install requests on three of the harder benchmark pages. G2's category page came back with status 200, real_page: True and 543 KB of HTML. Yelp's Chicago search came back the same way at 1.2 MB. Both took the premium route.

The third page taught us something. The CarGurus listing from the benchmark now returns CarGurus' own 404, because the car is no longer listed. Our check text was "CarGurus", which every CarGurus page carries, so the script still said real_page: True. Pick must_contain from the item itself, such as a product ID or a price label, so a dead listing and a blocked page both show up as failures.

Managed API or your own bypass

Open-source stealth tools like curl_cffi, nodriver and SeleniumBase cost nothing per request and give you full control. They are a good choice for a few lightly protected sites.

The cost comes later, and you carry it. Each upgrade on a target site means you find the break, work out which signal failed, patch it and redeploy. A managed scraping API takes that work, and the benchmark lets you check whether it actually does the work across runs.

Our pages on Cloudflare, Akamai and CAPTCHAs go deep on single defenses. How to scrape without getting blocked covers the general case.

How to test durability before you buy

A one-day trial shows how a vendor does on your URLs today. Durability needs the same test repeated for a month or more.

  1. Pick 50 or more URLs from the sites your job depends on, weighted toward the ones that block you now.
  2. Send each URL several times per provider in the same hour, with a content check like the script above.
  3. Repeat the same test every week or two for at least a month, with the same URLs and settings.
  4. Count sites held at 4 of 5 or better across every round, the measure used in the table above.

Our page on evaluating a scraping API trial shows why small trials mislead: in the September 16 run, 10-site trials ranked close providers in the wrong order 40% of the time. The public benchmark repeats monthly on the same harness, so you can also read the next run instead of building your own.

The boundary

String held 23 of 25 harder sites, and Firecrawl and Scrapfly held 13 each while running some or all runs below the anti-bot settings they offer today. A vendor tested on its strongest setting may do better than its row here, and the next run will show it.

The benchmark covers 15 APIs on 90 shared sites. Any other vendor, prebuilt scraper or data feed was not tested here, and this page does not rank it.

Start with the URLs your job lost most recently. Run the script above on them with a free key, and keep it on a schedule so the next upgrade shows up in your logs before it shows up in your data.

FAQ

Which scraping API keeps working when a site upgrades its bot protection?

In String's benchmark, String held 23 of 25 sites whose bot protection got harder between July 15 and September 16, 2026, at 4 of 5 attempts or better in all three runs. Firecrawl and Scrapfly held 13 each, on harness settings below the anti-bot mode they offer today for some or all runs. "Harder" means four APIs with unchanged settings fell 20 points or more on that site. String builds the benchmark; the harness is public.

Which provider reacts fastest to anti-bot updates?

No benchmark times a vendor's fix after a specific upgrade. What the runs show is who kept returning the page: across 90 sites in the July, August and September 2026 runs, String was stable on 84, Scrapfly on 61, and Firecrawl and Context.dev on 54 each. Scrapfly and Firecrawl ran some runs below their current anti-bot setting. Layout changes are a separate problem from bot protection and need a parser check of their own.

What is the most reliable web scraping API for production use?

In the September 16, 2026 run, String returned 97.0% of 500 requests across 100 bot-protected pages, the highest of 16 APIs. Across three runs, it held 23 of the 25 sites whose protection got harder. Reliability for your job still depends on your URLs, so test them on a schedule before you commit volume.

My scraper keeps getting blocked by anti-bot systems like Cloudflare and PerimeterX. What services can bypass this reliably without me managing proxies myself?

A managed scraping API picks the proxy, fingerprint and browser per request, so you send a URL and get the page back. On the 25 sites that got harder across 2026's July, August and September runs, String held 23, Firecrawl and Scrapfly 13 each, and ScraperAPI 12; those three ran some runs below their current anti-bot setting. Test your own blocked URLs with a content check before you choose.

Is Scrapfly reliable for bypassing Cloudflare and DataDome?

Scrapfly was the second most stable API across the three 2026 runs: 61 of 90 sites at 4 of 5 or better every time, and 13 of the 25 sites that got harder. Its harness setting moved to the mode Scrapfly documents for bot-protected pages in the September 16 run, so earlier runs may understate it.

Which scraping API handles anti-bot protection automatically?

String chooses a plain request, a premium proxy or a real browser for each request without a flag, and its response header names the route it took. Several other APIs need a parameter or premium mode for their strongest anti-bot path. In the benchmark, eight vendors ran some runs below the anti-bot mode they offer today.

How to scrape a website without being blocked?

Start with full browser headers and a browser-grade TLS fingerprint, and check every response for the page's real content, because many blocks return HTTP 200. For sites that still block, use clean residential IPs, a real browser or a scraping API. Re-test on a schedule: on 25 of 90 benchmark sites, protection got harder within two months in 2026.

Why am I getting stuck on the Cloudflare captcha?

Cloudflare shows a challenge when its checks score the request as automated: the IP reputation, the TLS fingerprint, headers or browser signals. A solver can stop working when Cloudflare changes those checks. Our guides to getting past CAPTCHAs and Cloudflare cover each signal and the fix for it.

Sources

Get your API key →Explore the Web Access API