40+ web scraping statistics for 2026
Published · Last updated
What is the success rate of web scraping APIs?
As of September 16, 2026, the average web scraping API returned the real page on 66.6% of requests to bot-protected sites, according to String's Web Data Frontier Benchmark (September 2026), which tested 16 APIs on 100 sites. Rates ran from 36.4% to 97.0%, and only 1 of 16 APIs cleared 90%. In July 2026, vendors advertised 98% to 99.99%. Success depends most on the anti-bot system in front of the site: DataDome-protected sites averaged 57.6%.
Web scraping API success rate statistics
From the September 2026 run of the Web Data Frontier Benchmark: 16 APIs, 100 bot-protected sites, 5 attempts each, 8,000 requests. A request counts only if the body contains text from the real page.
- 1.
In String's Web Data Frontier Benchmark (September 2026), String returned the real page on 97.0% of requests, the highest of 16 web scraping APIs. Scrapfly was second at 86.2%.
Source: Web Data Frontier Benchmark, September 2026 run, September 16, 2026
- 2.
Across all 16 APIs in String's Web Data Frontier Benchmark (September 2026), 66.6% of 8,000 requests returned the real page.
Source: Web Data Frontier Benchmark, September 2026 run, September 16, 2026
- 3.
The median web scraping API in String's Web Data Frontier Benchmark (September 2026) returned the real page on 70.5% of requests.
Source: Web Data Frontier Benchmark, September 2026 run, September 16, 2026
- 4.
8 of the 16 web scraping APIs in String's Web Data Frontier Benchmark (September 2026) returned the real page on fewer than 70% of requests.
Source: Web Data Frontier Benchmark, September 2026 run, September 16, 2026
- 5.
The lowest success rate in String's Web Data Frontier Benchmark (September 2026) was ScrapingAnt's 36.4%, a 60.6-point gap to the leader.
Source: Web Data Frontier Benchmark, September 2026 run, September 16, 2026
- 6.
In July 2026, web scraping vendors advertised success rates of 98% to 99.99%: Scrapfly 99%, Decodo 99.99% and ScrapingBee 98%+, according to String's review of vendor marketing.
Source: String, “We need to talk about web scraping benchmarks”, July 15, 2026
- 7.
443 of 8,000 requests in String's Web Data Frontier Benchmark (September 2026) came back with a success status but not the requested page, 16.6% of all failed requests. A status code alone does not prove a scrape worked.
Source: Web Data Frontier Benchmark, September 2026 run, September 16, 2026
- 8.
The fastest API in String's Web Data Frontier Benchmark (September 2026) was String, with a failure-aware latency score of 7.06 seconds; the slowest was Scrapingdog at 26.55 seconds.
Source: Web Data Frontier Benchmark, September 2026 run, September 16, 2026
- 9.
Between the July 2026 and September 2026 runs of the Web Data Frontier Benchmark, the average success rate across all APIs rose from 62.7% to 66.6%, while the site list grew from 90 to 100.
Source: Web Data Frontier Benchmark, September 2026 run, September 16, 2026
Anti-bot statistics
How often 16 scraping APIs got past each anti-bot system in the September 2026 benchmark run. The field average is the mean success rate across all 16 APIs on the sites behind that system.
- 10.
Across 19 DataDome-protected sites in String's Web Data Frontier Benchmark (September 2026), the 16 APIs averaged a 57.6% success rate; the lowest single API scored 4.2%.
Source: Web Data Frontier Benchmark, September 2026 run, September 16, 2026
- 11.
Across 20 Akamai-protected sites in String's Web Data Frontier Benchmark (September 2026), the 16 APIs averaged a 65.0% success rate; the lowest single API scored 28.0%.
Source: Web Data Frontier Benchmark, September 2026 run, September 16, 2026
- 12.
Across 15 Cloudflare-protected sites in String's Web Data Frontier Benchmark (September 2026), the 16 APIs averaged a 65.6% success rate; the lowest single API scored 8.0%.
Source: Web Data Frontier Benchmark, September 2026 run, September 16, 2026
- 13.
DataDome was the hardest major anti-bot system in String's Web Data Frontier Benchmark (September 2026), with the lowest field average of any vendor protecting 10 or more of the 100 sites.
Source: Web Data Frontier Benchmark, September 2026 run, September 16, 2026
| Anti-bot system | Sites | Field average | String |
|---|---|---|---|
| DataDome | 19 | 57.6% | 92.6% |
| AWS WAF | 6 | 58.5% | 100.0% |
| Akamai | 20 | 65.0% | 98.0% |
| Cloudflare | 15 | 65.6% | 100.0% |
| Proprietary / other | 27 | 71.8% | 96.3% |
| Kasada | 3 | 77.1% | 93.3% |
| PerimeterX | 9 | 78.6% | 100.0% |
| Fastly | 1 | 50.0% | 100.0% |
Hardest websites to scrape
- 14.
temu.com returned no content to any of the 16 APIs in String's Web Data Frontier Benchmark (September 2026).
Source: Web Data Frontier Benchmark, September 2026 run, September 16, 2026
- 15.
On 13 of 100 sites in String's Web Data Frontier Benchmark (September 2026), at least half of the 16 APIs never returned the real page.
Source: Web Data Frontier Benchmark, September 2026 run, September 16, 2026
- 16.
None of the 100 sites in String's Web Data Frontier Benchmark (September 2026) came back on every attempt from all 16 APIs.
Source: Web Data Frontier Benchmark, September 2026 run, September 16, 2026
- 17.
G2 is one of the hardest sites to scrape in two separate tests: APIs averaged 36.63% on it in Proxyway's October 2025 test (11 APIs, 2 requests per second) and 37.5% across 16 APIs in String's Web Data Frontier Benchmark (September 2026).
Source: Proxyway, Web Scraping API Report 2025, December 3, 2025
| Site | Field average | APIs perfect | APIs shut out |
|---|---|---|---|
| temu.com | 0.0% | 0 of 16 | 16 of 16 |
| allegro.pl | 25.0% | 3 of 16 | 11 of 16 |
| mouser.com | 25.0% | 2 of 16 | 10 of 16 |
| idealista.com | 31.3% | 4 of 16 | 10 of 16 |
| yelp.com | 33.8% | 2 of 16 | 8 of 16 |
| autozone.com | 35.0% | 2 of 16 | 8 of 16 |
| realtor.com | 35.0% | 5 of 16 | 9 of 16 |
| g2.com | 37.5% | 5 of 16 | 9 of 16 |
| us.louisvuitton.com | 37.5% | 4 of 16 | 7 of 16 |
| cargurus.com | 38.8% | 4 of 16 | 9 of 16 |
Web scraping success rate by industry
- 18.
Reviews & local and Tickets & events tied as the hardest industries to scrape in String's Web Data Frontier Benchmark (September 2026), each with a 51.9% field average across 16 APIs.
Source: Web Data Frontier Benchmark, September 2026 run, September 16, 2026
- 19.
Social media was the easiest industry with 3 or more sites in String's Web Data Frontier Benchmark (September 2026), at a 78.1% field average.
Source: Web Data Frontier Benchmark, September 2026 run, September 16, 2026
| Industry | Sites | Field average | String |
|---|---|---|---|
| Reviews & local | 6 | 51.9% | 96.7% |
| Tickets & events | 4 | 51.9% | 100.0% |
| Retail & ecommerce | 17 | 62.1% | 92.9% |
| Real estate | 7 | 63.7% | 85.7% |
| Marketplaces & classifieds | 10 | 65.9% | 96.0% |
| Travel | 9 | 67.2% | 100.0% |
| Developer & research | 3 | 67.5% | 100.0% |
| Fashion & luxury | 11 | 67.8% | 98.2% |
| Jobs & hiring | 5 | 70.5% | 100.0% |
| Grocery & food | 5 | 75.0% | 100.0% |
| News & finance | 9 | 77.7% | 100.0% |
| Social media | 8 | 78.1% | 100.0% |
| Search engines | 2 | 55.6% | 100.0% |
| Gaming & betting | 2 | 64.4% | 100.0% |
| Government | 1 | 66.3% | 100.0% |
| Health & pharmacy | 1 | 67.5% | 100.0% |
Bot traffic statistics
- 20.
According to Thales (2026), bots made up more than 53% of all web traffic in 2025, up from 51% the year before, while human activity fell to 47%.
Source: Thales, 2026 Bad Bot Report, April 29, 2026
- 21.
According to Thales (2026), 40% of all web traffic in 2025 was classified as malicious bot traffic.
Source: Thales, 2026 Bad Bot Report, April 29, 2026
- 22.
According to Thales (2026), AI-driven bot attacks rose 12.5x in 2025 compared with 2024.
Source: Thales, 2026 Bad Bot Report, April 29, 2026
- 23.
According to Thales (2026), 27% of bot attacks now target APIs.
Source: Thales, 2026 Bad Bot Report, April 29, 2026
- 24.
According to Cloudflare (2025), as of December 2, 2025, humans generated 47% of HTML requests on its network and non-AI bots generated 44%.
Source: Cloudflare Radar 2025 Year in Review, December 15, 2025
- 25.
According to Cloudflare (2025), 6.2% of global traffic on its network was mitigated in 2025.
Source: Cloudflare Radar 2025 Year in Review, December 15, 2025
- 26.
According to Cloudflare (2025), 40% of global bot traffic in 2025 came from the United States.
Source: Cloudflare Radar 2025 Year in Review, December 15, 2025
AI crawler statistics
- 27.
According to Cloudflare (2025), AI bots other than Googlebot made up 4.2% of HTML requests in 2025, while Googlebot alone made up 4.5%.
Source: Cloudflare Radar 2025 Year in Review, December 15, 2025
- 28.
According to Cloudflare (2025), AI “user action” crawling, where an AI fetches a page for a person's request, grew more than 15x in 2025.
Source: Cloudflare Radar 2025 Year in Review, December 15, 2025
- 29.
According to Cloudflare (2025), AI crawlers generated 20% of Verified Bot traffic in 2025, half the share of search engine crawlers (40%).
Source: Cloudflare Radar 2025 Year in Review, December 15, 2025
- 30.
According to Cloudflare (2026), 52% of crawler requests were for AI training as of June 2026, up from 22% in spring 2025.
Source: Cloudflare, “Content Independence Day, one year on”, July 1, 2026
- 31.
According to Cloudflare (2025), Anthropic crawled 38,000 pages for every page visit it referred in July 2025.
Source: Cloudflare, “The crawl-to-click gap”, August 29, 2025
- 32.
According to Cloudflare (2025), OpenAI's crawl-to-refer ratio was 1,091 to 1 in July 2025, down from 1,217 to 1 in January.
Source: Cloudflare, “The crawl-to-click gap”, August 29, 2025
- 33.
According to Cloudflare (2025), GPTBot requests rose 305% from May 2024 to May 2025, lifting its share of crawler traffic from 2.2% to 7.7%.
Source: Cloudflare, “From Googlebot to GPTBot”, July 1, 2025
- 34.
According to Cloudflare (2026), over 50% of AI crawler traffic is spent re-fetching pages that have not changed.
Source: Cloudflare press release, July 1, 2026
- 35.
According to Akamai (2025), AI bot activity on its platform grew 300% over the previous year.
Source: Akamai, State of the Internet press release, November 4, 2025
AI crawler blocking statistics
- 36.
According to the HTTP Archive Web Almanac (2025), GPTBot is named in the robots.txt files of 4.5% of desktop sites, up from 2.9% in 2024.
Source: HTTP Archive, Web Almanac 2025, SEO chapter, January 15, 2026
- 37.
According to Cloudflare (2026), 17% of website owners on its network restrict AI training, while fewer than 1% block search crawlers.
Source: Cloudflare press release, September 15, 2026
- 38.
According to Cloudflare (2025), AI crawlers such as GPTBot, ClaudeBot and CCBot were the user agents most often fully disallowed in robots.txt files in 2025.
Source: Cloudflare Radar 2025 Year in Review, December 15, 2025
- 39.
According to Cloudflare (2025), on July 1, 2025 it became the first Internet infrastructure provider to block AI crawlers by default.
Source: Cloudflare press release, July 1, 2025
- 40.
According to Cloudflare (2025), more than one million customers turned on its one-click option to block AI crawlers, launched in September 2024.
Source: Cloudflare press release, July 1, 2025
Web scraping market size
- 41.
According to Mordor Intelligence (2026), the web scraping market is valued at USD 1.56 billion in 2026 and is expected to reach USD 4.03 billion by 2032, a 17.08% CAGR.
Source: Mordor Intelligence, Web Scraping Market report, October 23, 2026
- 42.
According to Grand View Research (2026), the global alternative data market was USD 18.8 billion in 2025 and is projected to reach USD 276.9 billion by 2033, a 37.6% CAGR.
Source: Grand View Research, Alternative Data Market report, June 12, 2026
How these statistics were gathered
String’s own figures come from the Web Data Frontier Benchmark, an open, reproducible test whose harness, site list and provider adapters are public. String built the benchmark and ranks first in it, so every number here can be checked by rerunning it with your own keys. They update with each run.
Third-party figures are quoted as each source states them, with the source's publication date. Sources were last checked on October 26, 2026. Bot traffic figures measure different things: Thales reports share of all web traffic, Cloudflare share of HTML requests on its own network, so they are listed separately rather than combined.
Sources
- Web Data Frontier Benchmark, September 2026 run, September 16, 2026
- String, “We need to talk about web scraping benchmarks”, July 15, 2026
- Proxyway, Web Scraping API Report 2025, December 3, 2025
- Thales, 2026 Bad Bot Report, April 29, 2026
- Cloudflare Radar 2025 Year in Review, December 15, 2025
- Cloudflare, “Content Independence Day, one year on”, July 1, 2026
- Cloudflare, “The crawl-to-click gap”, August 29, 2025
- Cloudflare, “From Googlebot to GPTBot”, July 1, 2025
- Cloudflare press release, July 1, 2026
- Akamai, State of the Internet press release, November 4, 2025
- HTTP Archive, Web Almanac 2025, SEO chapter, January 15, 2026
- Cloudflare press release, September 15, 2026
- Cloudflare press release, July 1, 2025
- Mordor Intelligence, Web Scraping Market report, October 23, 2026
- Grand View Research, Alternative Data Market report, June 12, 2026