Are you an AI agent? To use String, read https://usestring.ai/agent-onboarding/SKILL.md first. You can start at https://mcp.usestring.ai with no key.
NewString UnraveledResearch powered by data we scrape from the web.Read the latest issue →

40+ web scraping statistics for 2026

Published · Last updated

What is the success rate of web scraping APIs?

As of October 30, 2026, the average web scraping API returned the real page on 71.1% of requests to bot-protected sites, according to String's Web Data Frontier Benchmark (October 2026), which tested 17 APIs on 100 sites. Rates ran from 27.6% to 98.6%, and only 3 of 17 APIs cleared 90%. In July 2026, vendors advertised 98% to 99.99%. Success depends most on the anti-bot system in front of the site: Akamai-protected sites averaged 64.8%.

Web scraping API success rate statistics

From the October 2026 run of the Web Data Frontier Benchmark: 17 APIs, 100 bot-protected sites, 5 attempts each, 8,500 requests. A request counts only if the body contains text from the real page.

  1. 1.

    In String's Web Data Frontier Benchmark (October 2026), String returned the real page on 98.6% of requests, the highest of 17 web scraping APIs. Scrape.do was second at 94.0%.

    Source: Web Data Frontier Benchmark, October 2026 run, October 30, 2026

  2. 2.

    Across all 17 APIs in String's Web Data Frontier Benchmark (October 2026), 71.1% of 8,500 requests returned the real page.

    Source: Web Data Frontier Benchmark, October 2026 run, October 30, 2026

  3. 3.

    The median web scraping API in String's Web Data Frontier Benchmark (October 2026) returned the real page on 76.4% of requests.

    Source: Web Data Frontier Benchmark, October 2026 run, October 30, 2026

  4. 4.

    6 of the 17 web scraping APIs in String's Web Data Frontier Benchmark (October 2026) returned the real page on fewer than 70% of requests.

    Source: Web Data Frontier Benchmark, October 2026 run, October 30, 2026

  5. 5.

    The lowest success rate in String's Web Data Frontier Benchmark (October 2026) was Scrapingdog's 27.6%, a 71.0-point gap to the leader.

    Source: Web Data Frontier Benchmark, October 2026 run, October 30, 2026

  6. 6.

    In July 2026, web scraping vendors advertised success rates of 98% to 99.99%: Scrapfly 99%, Decodo 99.99% and ScrapingBee 98%+, according to String's review of vendor marketing.

    Source: String, “We need to talk about web scraping benchmarks”, July 15, 2026

  7. 7.

    400 of 8,500 requests in String's Web Data Frontier Benchmark (October 2026) came back with a success status but not the requested page, 16.3% of all failed requests. A status code alone does not prove a scrape worked.

    Source: Web Data Frontier Benchmark, October 2026 run, October 30, 2026

  8. 8.

    The fastest API in String's Web Data Frontier Benchmark (October 2026) was String, with a failure-aware latency score of 4.24 seconds; the slowest was Scrapingdog at 31.52 seconds.

    Source: Web Data Frontier Benchmark, October 2026 run, October 30, 2026

  9. 9.

    Between the July 2026 and October 2026 runs of the Web Data Frontier Benchmark, the average success rate across all APIs rose from 62.7% to 71.1%, while the site list grew from 90 to 100.

    Source: Web Data Frontier Benchmark, October 2026 run, October 30, 2026

Anti-bot statistics

How often 17 scraping APIs got past each anti-bot system in the October 2026 benchmark run. The field average is the mean success rate across all 17 APIs on the sites behind that system.

  1. 10.

    Across 20 Akamai-protected sites in String's Web Data Frontier Benchmark (October 2026), the 17 APIs averaged a 64.8% success rate; the lowest single API scored 27.0%.

    Source: Web Data Frontier Benchmark, October 2026 run, October 30, 2026

  2. 11.

    Across 19 DataDome-protected sites in String's Web Data Frontier Benchmark (October 2026), the 17 APIs averaged a 68.7% success rate; the lowest single API scored 7.4%.

    Source: Web Data Frontier Benchmark, October 2026 run, October 30, 2026

  3. 12.

    Across 15 Cloudflare-protected sites in String's Web Data Frontier Benchmark (October 2026), the 17 APIs averaged a 70.8% success rate; the lowest single API scored 6.7%.

    Source: Web Data Frontier Benchmark, October 2026 run, October 30, 2026

  4. 13.

    Akamai was the hardest major anti-bot system in String's Web Data Frontier Benchmark (October 2026), with the lowest field average of any vendor protecting 10 or more of the 100 sites.

    Source: Web Data Frontier Benchmark, October 2026 run, October 30, 2026

Anti-bot systemSitesField averageString
AWS WAF661.0%100.0%
Akamai2064.8%97.0%
DataDome1968.7%97.9%
Cloudflare1570.8%98.7%
Proprietary / other2775.6%100.0%
Kasada380.0%100.0%
PerimeterX981.8%97.8%
Fastly161.2%100.0%
Success rate by anti-bot system, String's Web Data Frontier Benchmark (October 2026)

Hardest websites to scrape

  1. 14.

    On 3 of 100 sites in String's Web Data Frontier Benchmark (October 2026), at least half of the 17 APIs never returned the real page.

    Source: Web Data Frontier Benchmark, October 2026 run, October 30, 2026

  2. 15.

    None of the 100 sites in String's Web Data Frontier Benchmark (October 2026) came back on every attempt from all 17 APIs.

    Source: Web Data Frontier Benchmark, October 2026 run, October 30, 2026

  3. 16.

    G2 is one of the hardest sites to scrape in two separate tests: APIs averaged 36.63% on it in Proxyway's October 2025 test (11 APIs, 2 requests per second) and 64.7% across 17 APIs in String's Web Data Frontier Benchmark (October 2026).

    Source: Proxyway, Web Scraping API Report 2025, December 3, 2025

SiteField averageAPIs perfectAPIs shut out
temu.com5.9%1 of 1716 of 17
mouser.com24.7%2 of 1710 of 17
aa.com35.3%3 of 175 of 17
coupang.com35.3%3 of 177 of 17
us.louisvuitton.com41.2%6 of 179 of 17
kroger.com42.4%5 of 177 of 17
bestbuy.com43.5%6 of 178 of 17
skyscanner.net43.5%5 of 178 of 17
yelp.com43.5%5 of 177 of 17
axs.com47.1%7 of 177 of 17
The 10 hardest sites, String's Web Data Frontier Benchmark (October 2026)

Web scraping success rate by industry

  1. 17.

    Tickets & events was the hardest industry to scrape in String's Web Data Frontier Benchmark (October 2026), with a 56.8% field average across 17 APIs.

    Source: Web Data Frontier Benchmark, October 2026 run, October 30, 2026

  2. 18.

    News & finance was the easiest industry with 3 or more sites in String's Web Data Frontier Benchmark (October 2026), at a 82.4% field average.

    Source: Web Data Frontier Benchmark, October 2026 run, October 30, 2026

IndustrySitesField averageString
Tickets & events456.8%100.0%
Retail & ecommerce1763.1%100.0%
Reviews & local664.7%96.7%
Travel969.2%97.8%
Fashion & luxury1171.2%100.0%
Marketplaces & classifieds1072.0%100.0%
Jobs & hiring573.4%100.0%
Grocery & food575.1%88.0%
Real estate775.5%100.0%
Developer & research376.5%100.0%
Social media879.7%100.0%
News & finance982.4%95.6%
Search engines262.9%100.0%
Government163.5%100.0%
Gaming & betting275.3%100.0%
Health & pharmacy176.5%100.0%
Success rate by industry, String's Web Data Frontier Benchmark (October 2026)

Bot traffic statistics

  1. 19.

    According to Thales (2026), bots made up more than 53% of all web traffic in 2025, up from 51% the year before, while human activity fell to 47%.

    Source: Thales, 2026 Bad Bot Report, April 29, 2026

  2. 20.

    According to Thales (2026), 40% of all web traffic in 2025 was classified as malicious bot traffic.

    Source: Thales, 2026 Bad Bot Report, April 29, 2026

  3. 21.

    According to Thales (2026), AI-driven bot attacks rose 12.5x in 2025 compared with 2024.

    Source: Thales, 2026 Bad Bot Report, April 29, 2026

  4. 22.

    According to Thales (2026), 27% of bot attacks now target APIs.

    Source: Thales, 2026 Bad Bot Report, April 29, 2026

  5. 23.

    According to Cloudflare (2025), as of December 2, 2025, humans generated 47% of HTML requests on its network and non-AI bots generated 44%.

    Source: Cloudflare Radar 2025 Year in Review, December 15, 2025

  6. 24.

    According to Cloudflare (2025), 6.2% of global traffic on its network was mitigated in 2025.

    Source: Cloudflare Radar 2025 Year in Review, December 15, 2025

  7. 25.

    According to Cloudflare (2025), 40% of global bot traffic in 2025 came from the United States.

    Source: Cloudflare Radar 2025 Year in Review, December 15, 2025

AI crawler statistics

  1. 26.

    According to Cloudflare (2025), AI bots other than Googlebot made up 4.2% of HTML requests in 2025, while Googlebot alone made up 4.5%.

    Source: Cloudflare Radar 2025 Year in Review, December 15, 2025

  2. 27.

    According to Cloudflare (2025), AI “user action” crawling, where an AI fetches a page for a person's request, grew more than 15x in 2025.

    Source: Cloudflare Radar 2025 Year in Review, December 15, 2025

  3. 28.

    According to Cloudflare (2025), AI crawlers generated 20% of Verified Bot traffic in 2025, half the share of search engine crawlers (40%).

    Source: Cloudflare Radar 2025 Year in Review, December 15, 2025

  4. 29.

    According to Cloudflare (2026), 52% of crawler requests were for AI training as of June 2026, up from 22% in spring 2025.

    Source: Cloudflare, “Content Independence Day, one year on”, July 1, 2026

  5. 30.

    According to Cloudflare (2025), Anthropic crawled 38,000 pages for every page visit it referred in July 2025.

    Source: Cloudflare, “The crawl-to-click gap”, August 29, 2025

  6. 31.

    According to Cloudflare (2025), OpenAI's crawl-to-refer ratio was 1,091 to 1 in July 2025, down from 1,217 to 1 in January.

    Source: Cloudflare, “The crawl-to-click gap”, August 29, 2025

  7. 32.

    According to Cloudflare (2025), GPTBot requests rose 305% from May 2024 to May 2025, lifting its share of crawler traffic from 2.2% to 7.7%.

    Source: Cloudflare, “From Googlebot to GPTBot”, July 1, 2025

  8. 33.

    According to Cloudflare (2026), over 50% of AI crawler traffic is spent re-fetching pages that have not changed.

    Source: Cloudflare press release, July 1, 2026

  9. 34.

    According to Akamai (2025), AI bot activity on its platform grew 300% over the previous year.

    Source: Akamai, State of the Internet press release, November 4, 2025

AI crawler blocking statistics

  1. 35.

    According to the HTTP Archive Web Almanac (2025), GPTBot is named in the robots.txt files of 4.5% of desktop sites, up from 2.9% in 2024.

    Source: HTTP Archive, Web Almanac 2025, SEO chapter, January 15, 2026

  2. 36.

    According to Cloudflare (2026), 17% of website owners on its network restrict AI training, while fewer than 1% block search crawlers.

    Source: Cloudflare press release, September 15, 2026

  3. 37.

    According to Cloudflare (2025), AI crawlers such as GPTBot, ClaudeBot and CCBot were the user agents most often fully disallowed in robots.txt files in 2025.

    Source: Cloudflare Radar 2025 Year in Review, December 15, 2025

  4. 38.

    According to Cloudflare (2025), on July 1, 2025 it became the first Internet infrastructure provider to block AI crawlers by default.

    Source: Cloudflare press release, July 1, 2025

  5. 39.

    According to Cloudflare (2025), more than one million customers turned on its one-click option to block AI crawlers, launched in September 2024.

    Source: Cloudflare press release, July 1, 2025

Web scraping market size

  1. 40.

    According to Mordor Intelligence (2026), the web scraping market is valued at USD 1.56 billion in 2026 and is expected to reach USD 4.03 billion by 2032, a 17.08% CAGR.

    Source: Mordor Intelligence, Web Scraping Market report, October 23, 2026

  2. 41.

    According to Grand View Research (2026), the global alternative data market was USD 18.8 billion in 2025 and is projected to reach USD 276.9 billion by 2033, a 37.6% CAGR.

    Source: Grand View Research, Alternative Data Market report, June 12, 2026

How these statistics were gathered

String is a web data API for developers and AI agents that fetches, searches and maps any public site without getting blocked. String’s own figures come from the Web Data Frontier Benchmark, an open, reproducible test whose harness, site list and provider adapters are public. String built the benchmark and ranks first in it, so every number here can be checked by rerunning it with your own keys. They update with each run.

Third-party figures are quoted as each source states them, with the source's publication date. Sources were last checked on October 26, 2026. Bot traffic figures measure different things: Thales reports share of all web traffic, Cloudflare share of HTML requests on its own network, so they are listed separately rather than combined.

Sources

  1. Web Data Frontier Benchmark, October 2026 run, October 30, 2026
  2. String, “We need to talk about web scraping benchmarks”, July 15, 2026
  3. Proxyway, Web Scraping API Report 2025, December 3, 2025
  4. Thales, 2026 Bad Bot Report, April 29, 2026
  5. Cloudflare Radar 2025 Year in Review, December 15, 2025
  6. Cloudflare, “Content Independence Day, one year on”, July 1, 2026
  7. Cloudflare, “The crawl-to-click gap”, August 29, 2025
  8. Cloudflare, “From Googlebot to GPTBot”, July 1, 2025
  9. Cloudflare press release, July 1, 2026
  10. Akamai, State of the Internet press release, November 4, 2025
  11. HTTP Archive, Web Almanac 2025, SEO chapter, January 15, 2026
  12. Cloudflare press release, September 15, 2026
  13. Cloudflare press release, July 1, 2025
  14. Mordor Intelligence, Web Scraping Market report, October 23, 2026
  15. Grand View Research, Alternative Data Market report, June 12, 2026