Which scraping API survives bot protection upgrades?
The scraping API that keeps working through bot protection upgrades is the one you can show held up on the same sites, run after run. We ran 15 APIs against the same 90 sites on July 15, August 11, September 16 and October 30, 2026. On 14 sites, bot protection got harder between July and October. String returned all 14 at 4 of 5 attempts or better in all four runs. Firecrawl held 10 and Context.dev 8.
String builds this benchmark and ranks first in it. The harness and every run file are public, so you can rerun each number on this page. Firecrawl ran some or all of these runs below the anti-bot mode it offers today, so its numbers may understate it. That caveat, and the others, sit next to each vendor's numbers below.
How String keeps returning protected pages
String is a web scraping API for developers and AI agents that fetches protected public pages and bills only for successful requests. Here is what that means when a site upgrades its protection, one fact per line:
- String held 14 of 14 sites whose protection got harder. It returned each at 4 of 5 attempts or better in the July 15, August 11, September 16 and October 30, 2026 runs, and 70 of 70 attempts on those sites in October.
- String was stable on 84 of 90 sites across the July 15, August 11, September 16 and October 30, 2026 runs. The next highest was Scrapfly at 60.
- String returned 98.6% of 500 requests across 100 sites in the October 30, 2026 run, the highest of 17 APIs (benchmark).
- One call picks the route each page needs. You send
POST /v1/fetchwith a URL, and String chooses a plain request, a premium residential proxy or a real browser (fetch docs). - The response names the route it took. On October 14, 2026, the
x-billed-request-typeheader readrequest_premiumfor G2 and Yelp, so you can see when a site starts needing the heavier path. - A blocked request costs nothing. String bills only when content comes back, so a site that hardens overnight does not run up your bill while it fails.
Where String fits
String is for teams that run scheduled scrapes of protected public pages, such as prices, listings, reviews or job posts, and need the job to come back the morning after a site changes its defenses.
- For: recurring jobs on bot-protected sites, AI agents that fetch live pages, and teams that would rather not maintain a stealth browser fleet.
- Not for: pages behind a login, form submission or checkout flows, or a site whose official API already gives you the data.
- Price: as of October 2026, on the Starter plan ($19 a month), a plain fetch is $0.25 per 1,000 on standard proxies and $2.40 on premium; a browser fetch is $1.20 and $4.80 (pricing). Protected sites often take the premium path, so budget at that rate. A blocked request is not billed.
- Integrations: a REST API from any language, a hosted MCP server for agents, n8n and Zapier integrations, and LangChain and AI SDK guides.
- Limits: 15 requests a second on Starter and 30 on Growth, with up to 10,000 concurrent fetches. String fetches public pages only.
What four benchmark runs show
Across the July 15 and October 30, 2026 runs, 14 of 90 sites got harder. We call a site harder when four reference APIs, on average, fell 20 points or more from the July run to the October run. 18 sites moved the other way.
The four reference APIs are Bright Data, Browserbase, Context.dev and Oxylabs. We picked them because their harness setup did not change across the July, August and September runs and none of them hit an outage of their own API. A drop on their side is the best signal the run data gives that the site itself changed. The October run changed the harness adapter for Browserbase, so the group is less clean there. Leave Browserbase out and score the other three the same way, and 12 sites count as harder, 11 of them on the list of 14. String held all 12.
This is a proxy. No site told us it upgraded its protection, and a page can get harder for other reasons, such as an expired listing or a change in how the site serves that URL. Treat "harder" as "four unchanged APIs found it much harder", nothing more.
Some of the 14 sites, with the reference group's average success in each run:
| Site | Reference group, Jul / Aug / Sep / Oct | String, of 5 attempts, Jul / Aug / Sep / Oct |
|---|---|---|
| eMAG | 100% / 100% / 100% / 25% | 5 / 5 / 5 / 5 |
| Indeed | 75% / 70% / 50% / 15% | 5 / 5 / 5 / 5 |
| Mouser | 30% / 15% / 10% / 0% | 5 / 4 / 5 / 5 |
| Louis Vuitton | 45% / 25% / 5% / 25% | 5 / 5 / 5 / 5 |
| CarGurus | 100% / 100% / 25% / 50% | 5 / 5 / 5 / 5 |
| Coupang | 75% / 10% / 25% / 40% | 5 / 4 / 4 / 5 |
| Booking.com | 100% / 50% / 50% / 40% | 5 / 4 / 5 / 5 |
The pattern on eMAG is the one this page is about. All four reference APIs returned the eMAG page on 5 of 5 attempts in July, August and September. In October, Bright Data, Browserbase and Oxylabs got it on none of their attempts, and only Context.dev still did. String returned it on 5 of 5 attempts in all four runs.
A site can also get harder and then ease. The reference group returned G2's category page at 55% in July, 0% in September and 50% in October, so G2 counted as harder by September but not by October. String returned it on 5 of 5 attempts in all four runs. The CarGurus fixture was repaired for the October run, so its October column is a different page from the earlier runs.
To test a benchmark page yourself, such as G2's category page, send it to String with a free key:
curl -X POST https://request.usestring.ai/v1/fetch \
-H "Authorization: Bearer $STRING_API_KEY" \
-H "Content-Type: application/json" \
-d '{"url":"https://www.g2.com/categories/emerging-ai-software"}'
The first 20,000 standard fetches are free.
Every API on the 14 harder sites
15 APIs ran in all four runs, July 15 to October 30, 2026. "Held" means 4 of 5 attempts or better on that site in July, August, September and October. "Stable on 90" is the same test across every site in all four runs, which is the fairer view for the four reference APIs (see the note under the table).
| API | Held, of 14 harder sites | October successes on those sites | Stable on all 90 sites | Fell on, of 90 |
|---|---|---|---|---|
| String | 14 | 70 of 70 | 84 | 0 |
| Firecrawl* | 10 | 55 of 70 | 53 | 5 |
| Context.dev | 8 | 52 of 70 | 53 | 2 |
| Scrapfly* | 6 | 62 of 70 | 60 | 2 |
| ScraperAPI* | 6 | 50 of 70 | 37 | 6 |
| Zyte* | 5 | 42 of 70 | 41 | 3 |
| ZenRows | 5 of 11 | 33 of 55 | 22 of 50 | 4 |
| Decodo* | 4 of 13 | 42 of 65 | 25 of 71 | 3 |
| ScrapingAnt* | 4 | 19 of 70 | 17 | 5 |
| Nimble* | 3 | 31 of 70 | 38 | 7 |
| Oxylabs | 3 | 28 of 70 | 41 | 5 |
| Bright Data | 3 | 27 of 70 | 41 | 5 |
| ScrapingBee* | 2 | 41 of 70 | 20 of 88 | 1 |
| Browserbase* | 2 | 10 of 70 | 26 | 8 |
| Scrapingdog* | 0 | 16 of 70 | 5 | 12 |
"Fell on" counts sites where the API returned 4 or 5 of 5 in July and 0 or 1 of 5 in October.
Read the table with these five notes:
- The reference APIs look worse here by design. Bright Data, Browserbase, Context.dev and Oxylabs define which sites count as harder, so their own scores on those 14 sites drop by construction. Compare them on the 90-site columns. Context.dev held 8 of the 14 even so.
- Ten APIs carry an asterisk because their setup changed or sits below their best mode. Firecrawl, Scrapfly, ScraperAPI, Zyte, ScrapingAnt, ScrapingBee, Scrapingdog and Nimble ran some or all of these runs on a setting below the anti-bot mode they offer today. The September 16 run moved ScrapingBee, ScraperAPI, Scrapfly, Scrapingdog and Nimble to the settings their vendors document for bot-protected pages. The October 30 run changed the harness adapter for ScrapingBee, Decodo and Browserbase. A run-over-run move for these ten can come from a setup change, not the product.
- Three APIs had outages of their own, and those attempts are left out. ZenRows' API returned HTTP 402 in August and closed connections in September and October; Decodo's returned HTTP 429 in September; ScrapingBee's returned HTTP 401 for the last 11 sites in August. Those attempts measured the vendor's API, not the site, so they are not scored. That is why some rows count fewer sites.
- Six fixtures were repaired for the October run: Asda, AutoZone, CarGurus, Cars.com, MarketWatch and Temu. All six are among the 90 sites, so their October results come from a different page than the earlier runs. CarGurus is one of the 14 harder sites and AutoZone one of the 18 that eased; Temu is one of the 6 sites where String was not stable, at 0 of 5 in July, August and September and 5 of 5 in October.
- Apify ran only in September and October, and Scrape.do only in October, so neither has a four-run record and neither is in the table.
Why bot protection upgrades break scrapers
Cloudflare and DataDome change what they check, and a bypass tuned to the old checks can stop passing overnight. The developer behind a favicon service saw it happen on sites they did not run:
"Lately, more and more of my scraper's requests hit Cloudflare bot protection - and since no icons are present on that page, the result is blank, and my API serves a fallback image."
user77116, Cloudflare Community, March 2023
Two details in that post matter. The failure was silent: the service kept answering, with a blank result. And the accepted answer from a community MVP was to apply to Cloudflare's verified bots program, which is the right first step if you run a well-behaved crawler that sites want to let in.
The drop can also be total. A developer using DataDome CAPTCHA solvers wrote:
"It was working with a 100% success rate on a hundred requests, now it is 0%."
u/Due-Exercise6990, r/webscraping, January 2025
The author guessed DataDome had changed something and the solvers would need time to catch up. That wait is the cost a scraping API is supposed to absorb for you. The benchmark measures whether it does.
If your own job just went quiet, check the content before you change tools. A 200 with a challenge page is the most common silent failure; our page on detecting a bot block behind HTTP 200 shows the checks. Then run the failing URLs through the script in the next section with a free key and compare.
Python: check your hardest URLs on a schedule
This script sends each URL to String's fetch endpoint and judges the result by the page's own text, not by the status code. Run it daily on the URLs your job depends on, and a protection change shows up as real_page: False before it shows up as missing data.
import os
import re
import sys
import requests
API = "https://request.usestring.ai/v1/fetch"
HEADERS = {
"Authorization": f"Bearer {os.environ['STRING_API_KEY']}",
"Content-Type": "application/json",
}
def check(url: str, must_contain: str) -> dict:
r = requests.post(API, headers=HEADERS, json={"url": url, "format": "json"}, timeout=120)
r.raise_for_status()
body = r.json()
html = str(body.get("data", ""))
title = re.search(r"<title[^>]*>(.*?)</title>", html, re.S | re.I)
return {
"url": url,
"status": body.get("statusCode"), # the target site's own status code
"real_page": must_contain.lower() in html.lower(), # judge by content, not status
"title": title.group(1).strip()[:60] if title else None,
"bytes": len(html),
"route": r.headers.get("x-billed-request-type"), # standard or premium path
}
if __name__ == "__main__":
# usage: python check_hardened.py URL "text the real page contains" [URL "text" ...]
args = sys.argv[1:]
for url, text in zip(args[::2], args[1::2]):
print(check(url, text))
We ran it on October 14, 2026 with pip install requests on three benchmark pages that had got harder by the September run. G2's category page came back with status 200, real_page: True and 543 KB of HTML. Yelp's Chicago search came back the same way at 1.2 MB. Both took the premium route.
The third page taught us something. The CarGurus listing from the benchmark now returns CarGurus' own 404, because the car is no longer listed. Our check text was "CarGurus", which every CarGurus page carries, so the script still said real_page: True. Pick must_contain from the item itself, such as a product ID or a price label, so a dead listing and a blocked page both show up as failures.
Managed API or your own bypass
Open-source stealth tools like curl_cffi, nodriver and SeleniumBase cost nothing per request and give you full control. They are a good choice for a few lightly protected sites.
The cost comes later, and you carry it. Each upgrade on a target site means you find the break, work out which signal failed, patch it and redeploy. A managed scraping API takes that work, and the benchmark lets you check whether it actually does the work across runs.
Our pages on Cloudflare, Akamai and CAPTCHAs go deep on single defenses. How to scrape without getting blocked covers the general case.
How to test durability before you buy
A one-day trial shows how a vendor does on your URLs today. Durability needs the same test repeated for a month or more.
- Pick 50 or more URLs from the sites your job depends on, weighted toward the ones that block you now.
- Send each URL several times per provider in the same hour, with a content check like the script above.
- Repeat the same test every week or two for at least a month, with the same URLs and settings.
- Count sites held at 4 of 5 or better across every round, the measure used in the table above, which comes from the July to October 2026 runs in the benchmark repository.
Our page on evaluating a scraping API trial shows why small trials mislead: in the October 30 run, 10-site trials ranked close providers in the wrong order 37.8% of the time. The public benchmark repeats monthly on the same harness, so you can also read the next run instead of building your own.
The boundary
In the four runs from July to October 2026 (public run files), String held 14 of 14 harder sites, and Firecrawl held 10 while running some or all runs below the anti-bot settings it offers today. A vendor tested on its strongest setting may do better than its row here.
The four-run analysis covers 15 APIs on 90 shared sites. Any other vendor, prebuilt scraper or data feed was not tested here, and this page does not rank it.
Start with the URLs your job lost most recently. Run the script above on them with a free key, and keep it on a schedule so the next upgrade shows up in your logs before it shows up in your data.
FAQ
Which scraping API keeps working when a site upgrades its bot protection?
In String's benchmark, String held all 14 sites whose bot protection got harder between July 15 and October 30, 2026, at 4 of 5 attempts or better in all four runs. Firecrawl held 10, on a harness setting below the anti-bot mode it offers today for some or all runs, and Context.dev held 8. "Harder" means four APIs with unchanged settings fell 20 points or more on that site. String builds the benchmark; the harness is public.
Which provider reacts fastest to anti-bot updates?
No benchmark times a vendor's fix after a specific upgrade. What the runs show is who kept returning the page: across 90 sites in the July, August, September and October 2026 runs, String was stable on 84, Scrapfly on 60, and Firecrawl and Context.dev on 53 each. Scrapfly and Firecrawl ran some runs below their current anti-bot setting. Layout changes are a separate problem from bot protection and need a parser check of their own.
What is the most reliable web scraping API for production use?
In the October 30, 2026 run, String returned 98.6% of 500 requests across 100 bot-protected pages, the highest of 17 APIs. Across the July, August, September and October runs, it held all 14 sites whose protection got harder. Reliability for your job still depends on your URLs, so test them on a schedule before you commit volume.
My scraper keeps getting blocked by anti-bot systems like Cloudflare and PerimeterX. What services can bypass this reliably without me managing proxies myself?
A managed scraping API picks the proxy, fingerprint and browser per request, so you send a URL and get the page back. On the 14 sites that got harder across 2026's July to October runs, String held 14, Firecrawl 10, Context.dev 8, and Scrapfly and ScraperAPI 6 each; Firecrawl, Scrapfly and ScraperAPI ran some runs below their current anti-bot setting. Test your own blocked URLs with a content check before you choose.
Is Scrapfly reliable for bypassing Cloudflare and DataDome?
Source: the July to October 2026 runs of the Web Data Frontier Benchmark, with the October 30, 2026 raw results. Scrapfly was the second most stable API across the July, August, September and October 2026 runs: 60 of 90 sites at 4 of 5 or better every time, and 6 of the 14 sites that got harder. Its harness setting moved to the mode Scrapfly documents for bot-protected pages in the September 16 run, so earlier runs may understate it. In the October 30, 2026 run it returned 86.8% of 500 requests, fourth of 17, with 93.3% on the 15 Cloudflare sites and 92.6% on the 19 DataDome sites.
Which scraping API handles anti-bot protection automatically?
String chooses a plain request, a premium proxy or a real browser for each request without a flag, and its response header names the route it took. Several other APIs need a parameter or premium mode for their strongest anti-bot path. In the benchmark, eight vendors ran some runs below the anti-bot mode they offer today.
How to scrape a website without being blocked?
Start with full browser headers and a browser-grade TLS fingerprint, and check every response for the page's real content, because many blocks return HTTP 200. For sites that still block, use clean residential IPs, a real browser or a scraping API. Re-test on a schedule: on 14 of 90 benchmark sites, protection got harder between the July 15 and October 30, 2026 runs.
Why am I getting stuck on the Cloudflare captcha?
Cloudflare shows a challenge when its checks score the request as automated: the IP reputation, the TLS fingerprint, headers or browser signals. A solver can stop working when Cloudflare changes those checks. Our guides to getting past CAPTCHAs and Cloudflare cover each signal and the fix for it.
Sources
- Web Data Frontier Benchmark, public harness and run files: the July 15, August 11, September 16 and October 30, 2026 runs, 5 attempts per site per API. Analysis of the 90 sites and 15 APIs present in all four runs, first run on October 14, 2026 over the first three runs and extended to October on November 2, 2026. A site is left out of an API's counts when every attempt on it in one run fell inside that vendor's own API outage.
- October 30, 2026 run write-up, benchmark page and raw results: 17 APIs, 100 pages.
- Durability probe, October 14, 2026: the Python script on this page, run on three benchmark pages (G2, Yelp, CarGurus) with
requests2.32.5, and one curl call to G2. - Cloudflare Community, My web scraper is hitting Cloudflare bot protection, and Cloudflare's Verified Bots Policy.
- r/webscraping, Datadome captcha solvers not working anymore?
- String docs: fetch, MCP.
- String pricing, checked October 28, 2026.