Guides

Web scraping statistics for 2026: API success rates from an open 100-site benchmark, anti-bot and hardest-site figures, bot traffic, AI crawlers and market size, each with its source and date.

We sent one request to each of 100 bot-protected pages from Node.js on October 20, 2026. Plain fetch got 40, fetch with Chrome headers got 38, got-scraping got 50 and String got 86. Eight steps, every code block run.

We dated the top 3 Google results for 78 questions: 30% were updated after June 2026, the newest model training cutoff, and 49% had no date in their metadata. Python to search, date each page and keep a RAG index fresh.

Python and TypeScript that ran on October 23, 2026, the 'Robot or human?' page Walmart returns as HTTP 200, and how 16 scraping APIs did on walmart.com in the September 16, 2026 benchmark.

We priced 99 protected pages at the rate each was billed. Premium proxies were 42% of requests and 79% of the bill. Six steps, with Python, to pay the high rate only where a page needs it.

One job, three runs, 99 protected pages: 94 came back every run. A Python job with a lock, a body check and an exit code, scheduled with cron, GitHub Actions or APScheduler.

How to give an agent web search: the tool schema, a handler in Python and TypeScript, and a fetch step. On 57 questions, full questions and keyword forms shared a median of 4 Google result domains.

n8n's HTTP Request node got 42 of 100 protected pages on its defaults and 45 with Chrome headers. The same node calling String got 90. An importable workflow that routes only blocked URLs.

Every Firecrawl parameter mapped to String, the scrape call swapped, and a fallback that keeps Firecrawl as primary: on the September 16, 2026 benchmark it lifted coverage from 80.2% to 97.2%.

Python and TypeScript that ran on October 12, 2026, what plain requests get from YouTube, and how 16 scraping APIs did on youtube.com in the September 16, 2026 benchmark.

A Yahoo Finance script that ran on October 12, 2026, the 429 that is not a rate limit, and how 16 scraping APIs did on six finance sites in the September 16, 2026 benchmark.

A Google Flights script that ran on October 7, 2026, the fare-card label it parses, and how 16 scraping APIs did on aa.com, delta.com, Kayak and Skyscanner in the September 16, 2026 benchmark.

What Cloudflare checks, what each block looks like, a DIY ladder with code we ran on eight Cloudflare sites, and how 16 scraping APIs did on 15 Cloudflare sites in the September 16, 2026 benchmark.

Nine techniques that still work in 2026, each with the point where it breaks: pacing, residential IPs, TLS matching, stealth browsers, sessions, soft blocks at HTTP 200, and when to stop.

Python and TypeScript that read Reddit's JSON listings and comment trees on the day this was written, the three ways a plain request fails, and how 16 scraping APIs did on reddit.com.

A Python rank tracker and a TypeScript top-30 pull that ran on the day this was written, why a plain request gets a page with no results, and how 16 scraping APIs did on Google.

Python and TypeScript that read public X profiles and posts on the day this was written, what X shows logged out, and how 16 scraping APIs did on X, Instagram and TikTok.

Python and TypeScript that ran against Zillow on the day this was written, what a plain request gets back, and how 16 scraping APIs did on zillow.com in the September 16, 2026 benchmark.

Python and TypeScript that ran against Amazon product and search pages on the day this was written, the HTTP 200 with no price to check for, and how 16 scraping APIs did on amazon.com.