NewLaunching String Web Access APIRead the manifesto →
← Comparisons

Best Web Scraping Tools 2026: 12 Compared, Free and Paid

Bruce Magness, Chief of Staff, String · Updated October 5, 2026

Last updated: October 5, 2026. Benchmark figures come from the September 16, 2026 run of the Web Data Frontier Benchmark. Pricing was checked against each vendor's published pages on October 5, 2026.

The best web scraping tools in 2026 are String, Scrapfly, ScraperAPI, Firecrawl, Bright Data, Context.dev and Oxylabs if you want a paid hosted API, Apify if you want a prebuilt scraper for a named site, Scrapy, Crawlee, Playwright or Crawl4AI if you want free open-source code you run yourself, and Octoparse if you do not write code. On 100 bot-protected sites, String returned content on 97.0% of requests, the highest of 16 providers tested, with the fastest latency score. Scrapfly placed second at 86.2%, ScraperAPI third at 84.0% and Firecrawl fourth at 80.2%. Apify's Web Fetch Actor scored 77.4%. The open-source tools and Octoparse were not tested, because a harness cannot call them the same way.

The choice starts with who carries the blocking problem. A free framework leaves proxies, browsers and captchas to you. A paid API charges per page and handles them. The benchmark measures the second group, so every rank on this page is an API rank.

Where String fits. Teams whose sites block plain clients and who need the page back, from scripts, pipelines or agents. Price: Starter is $20 a month and Growth $100 a month, and Growth bills $0.20 per 1,000 standard fetches and $2.00 on a premium proxy. Failed requests are not billed. It connects through a REST API, a hosted MCP server, Apify Actors and LangChain, and it returns Markdown, raw HTML, or the fields you name in a JSON Schema. Protected sites can need a browser and a premium proxy, which bill at the higher rates. If your sites are open and you want free code on your own servers, Scrapy or Crawl4AI does that job.

TL;DR

  • String passed 97.0% of benchmark requests, 485 of 500, first of 16 providers, with the fastest latency score at 7.06 seconds. Failed requests are free and the first 5,000 standard requests need no card.
  • Scrapfly passed 86.2%, ScraperAPI 84.0%, Firecrawl 80.2%, Apify's Web Fetch Actor 77.4%, Bright Data 74.6%, Context.dev 72.0% and Oxylabs 69.0%.
  • No other single API covers every site. Scrapfly and Firecrawl together returned all five attempts on 93 of 100 sites, the same count as String alone, at $199 a month in plans before usage.
  • Scrapy, Crawlee, Playwright, Puppeteer, Selenium and Crawl4AI are free and open source. None ships a proxy pool or a captcha solver. A Scrapy spider can keep its crawl and send only blocked pages to an API; the code is below.
  • Oxylabs vs Apify: Apify returned more attempts on 25 sites, Oxylabs on 17, and they tied on 58.
  • Apify sells prebuilt scrapers called Actors. Its Web Fetch Actor costs from $1.00 per 1,000 fetches, and unused prepaid usage expires each month.
  • Octoparse is the no-code option, from a free plan to Standard at $99 a month on its pricing cards.
  • Every benchmark number comes from a public harness you can rerun with your own API keys.

Twelve web scraping tools at a glance

Tool Type Best for Benchmark success Price near $100 a month Watch for
String Hosted API Protected sites that must return content 97.0% Growth, $100: $0.20 per 1,000 standard fetches, $2.00 premium Pair it with your own crawl loop: sitemap discovery, then fetch
Scrapfly Hosted API Several APIs on one key 86.2% Pro, $100: 1,000,000 credits; 1 to 25 credits a page by setting Credits do not roll over; failures bill if over 30% of an hour's traffic fails
ScraperAPI Hosted API Akamai-protected sites 84.0% Startup, $149: 1,000,000 credits; protected sites 10 credits each Bills 200 and 404 responses; no rollover
Firecrawl Hosted API Crawl-to-Markdown for LLMs 80.2% Standard, $99 billed monthly: $0.99 per 1,000 pages A page that returns a 403 or 404 still costs a credit
Apify Scraper marketplace A prebuilt scraper for a named site 77.4% (Web Fetch Actor) Starter $19 or Scale $199, plus usage; Web Fetch from $1.00 per 1,000 Each Store Actor has its own price and maintainer
Bright Data Hosted API Proxies and datasets on one contract 74.6% No plan near $100; $1.50 per 1,000 successful requests 47.4% against DataDome
Context.dev Hosted API One flat credit per page 72.0% Pro, $99: 125,000 credits, $0.79 per 1,000 pages 404s now bill at the base price
Oxylabs Hosted API Grocery and news sites 69.0% Web Unblocker Micro, $75 for 8GB Billed by bandwidth, so no per-page rate
Crawl4AI Open-source crawler LLM-ready Markdown on your own servers Not tested Free, Apache-2.0 No proxies or unblocking included
Scrapy Open-source framework Large crawls you control in Python Not tested Free, BSD-3-Clause Does not render JavaScript on its own
Playwright Open-source browser automation Pages that need clicks and scripts Not tested Free, Apache-2.0 A browser without a proxy pool is easy to block
Octoparse No-code desktop app Point-and-click scraping without code Not tested Standard, $99 a month on its pricing card Tasks and concurrency are capped by plan

Benchmark figures come from the September 16, 2026 run: 100 sites, five attempts each, 500 requests per provider. The latency score is failure-aware: per site, the p75 of a provider's successful attempts, and a site it never returned inherits the p75 of the providers that did. Prices come from each vendor's published pages, checked October 5, 2026, at the plan closest to $100 a month billed monthly.

Which kind of web scraping tool do you need?

There are four kinds, and they differ on one question: who handles blocking.

A hosted web scraping API takes a URL and returns the page. Proxy rotation, browser rendering and captcha handling happen on the vendor's side, and you pay per page or per credit. String, Scrapfly, ScraperAPI, Context.dev, Firecrawl, Bright Data and Oxylabs are this kind. It is the right choice when your sites are protected and your team would rather not run proxy infrastructure.

A scraper marketplace sells ready-made scrapers for specific sites. Apify is the example here, with more than 68,000 Actors in its store. It is the right choice when someone has already built and maintains a scraper for the exact site you need.

An open-source framework is code you run. Scrapy crawls at scale, Playwright drives a real browser, and Crawl4AI turns pages into Markdown for language models. They cost nothing to license. They are the right choice when your sites are unprotected, or when you already own proxies and want full control.

Seven open-source tools come up in every list of free scrapers. None of them was benchmarked, because each one's result depends on the proxies you add.

Tool Language License Renders JavaScript Proxies or captcha solving
Scrapy Python BSD-3-Clause Not by default; add a headless browser Bring your own
Crawlee TypeScript, and Python Apache-2.0 Yes, through Playwright or Puppeteer Rotates proxies you supply
Playwright TypeScript, Python, Java, .NET Apache-2.0 Yes Bring your own
Puppeteer JavaScript Apache-2.0 Yes Bring your own
Selenium Java, Python, C#, Ruby, JavaScript Apache-2.0 Yes Bring your own
Crawl4AI Python Apache-2.0 Yes Bring your own
Beautiful Soup Python MIT No, it parses HTML you already have Not applicable

A no-code app lets you click on the data you want. Octoparse is the example here. It is the right choice for a non-developer with a small, steady list of sites.

Many teams end up with two kinds: a framework or no-code app for easy sites, and an API for the ones that block.

How we tested the web scraping APIs

We ran 100 live sites, five attempts each, against 16 providers, for 500 requests per provider, on September 16, 2026. Each provider ran through one adapter published in the repository. The timeout was 90 seconds. A request passed only when the response contained a marker from the real page, so a captcha page returning HTTP 200 counted as a failure. The 100 sites sit behind DataDome, Akamai, Cloudflare, PerimeterX, AWS WAF, Kasada, Fastly and in-house systems.

Five adapters moved before this run to the settings their vendors document for bot-protected pages: Scrapfly, ScraperAPI, ScrapingBee, Scrapingdog and Nimble. Scrapfly ran with anti-scraping protection on its residential pool. Firecrawl ran its default proxy path, which its docs now bill at the same credit as the enhanced proxy. Oxylabs ran Web Unblocker without its JavaScript-render header. Apify ran its Web Fetch Actor, which sends every fetch through Apify Proxy's unblocker group.

Scrapy, Crawlee, Playwright, Puppeteer, Selenium, Crawl4AI, Beautiful Soup and Octoparse were not tested. They are not hosted services, so their success depends on the proxies and code you bring. Of Apify's Store, only the Web Fetch Actor was tested. None of the untested tools gets a rank on this page.

Disclosure: String built this benchmark and sells the API that ranks first in it. The answer to that conflict is a harness you can run yourself. The targets, pass rule and every adapter are open on GitHub and accept your own keys. The original benchmark post explains why the latency score refuses to reward fast failures. Providers contest results and we publish the corrections: Firecrawl's engineer opened a pull request against its adapter, and the rerun raised its score by 7.7 points.

Overall success rate across all 500 requests per provider. Web Data Frontier Benchmark · 100 targets · 500 requests per provider · September 16, 2026. Current leaderboard

String: the highest success rate and the fastest latency score

String's Web Access API makes one HTTP call against any URL. Proxy rotation, anti-bot handling, captcha solving and JavaScript rendering happen behind the endpoint.

Pricing. Usage-based. Growth costs $100 a month and bills $0.20 per 1,000 fetches on a standard proxy, $2.00 on a premium proxy, $1.00 for a browser request on a standard proxy, and $4.00 with both. Starter costs $20 a month at $0.30, $3.00, $1.50 and $6.00. The first 5,000 standard requests are free, blocked requests are not billed, and credits roll over. Protected sites can need a browser or a premium proxy, which bill at the higher rates.

Benchmark success. 97.0%, or 485 of 500, first of 16. All five attempts on 93 sites, partial content on 5, nothing on 2 (temu.com and idealista.com).

Latency. A latency score of 7.06 seconds, the fastest of 16.

Strengths. 100% against Cloudflare, PerimeterX, AWS WAF and Fastly, 98.0% against Akamai and 96.3% against in-house systems. A hosted MCP server gives an agent fetch, search and sitemap discovery without local setup, and a jsonSchema field returns structured JSON with only the fields you define.

Pair it with. A crawl loop of your own. String discovers a site's URLs through its sitemap endpoint and fetches each page; a framework such as Scrapy or Crawl4AI can drive the crawl and hand String the pages that block it.

Scrapfly: 86.2% and five APIs on one key

Pricing. Pro is $100 a month for 1,000,000 credits, with overflow at $3.50 per 10,000. A datacenter scrape is 1 credit, JavaScript rendering adds 5, and residential proxies are 25, so a page costs from $0.10 to $2.50 per 1,000 and more with rendering. Failed requests are free, except 404s and except when more than 30% of an hour's traffic fails, per its billing docs. Credits do not roll over. The free tier is 1,000 credits with no time limit.

Benchmark success. 86.2%, or 431 of 500, second of 16. Nothing returned on 5 sites (Best Buy, Google Search, Ralph Lauren, Temu and Zara), the fewest after String.

Latency. A latency score of 19.79 seconds, eleventh of 16. On the 67 sites where Scrapfly and String both passed every attempt, Scrapfly's median was 6.16 seconds against String's 1.66.

Strengths. Scraping, browser, crawler, extraction and screenshot APIs share one key and one credit pool, and it ships a hosted MCP server. It holds SOC 2 Type II, ISO 27001 and HIPAA attestations. 92.6% against DataDome and 96.7% against AWS WAF, and 100% on real-estate and social media sites.

Weaknesses. Speed on pages both return, and a bill that moves with the setting: its anti-bot mode can move a request to the residential pool, which is 25 times the datacenter rate. 20% on the one Fastly site.

ScraperAPI: 84.0% and the strongest Akamai record after String

Pricing. The plan nearest $100 is Startup at $149 a month for 1,000,000 API credits and 50 concurrent threads; Hobby is $49. A plain request is 1 credit. Premium proxies cost 10, ultra-premium 30, and requests to Cloudflare, DataDome, PerimeterX and Akamai sites cost 10 each, per its credit docs. It bills 200 and 404 responses. Credits do not roll over. The free plan is 1,000 credits, and a 7-day trial gives 5,000.

Benchmark success. 84.0%, or 420 of 500, third of 16. All five attempts on 71 sites, partial on 20, nothing on 9, among them Facebook, Instagram and X.

Latency. A latency score of 12.97 seconds, third of 16.

Strengths. 92.0% against Akamai, the best of the seven other APIs, and 100% on news and finance sites. An MCP server.

Weaknesses. 62.5% on social media. Its August score was 64.2%; the September adapter switched on the setting ScraperAPI documents for protected pages, so the jump measures our adapter, not a product change. See String vs ScraperAPI.

Context.dev: one flat credit per page

Pricing. Developer is $19 a month for 7,500 credits and Pro is $99 for 125,000, or $0.79 per 1,000 pages, with overage at $1.80 per 1,000. One credit is one scrape, with proxies, rendering and anti-bot bypass included. Browser actions cost 2 credits. Failed requests are not billed, but a 404 bills at the base price. Work-email accounts get 1,000 free credits a month; personal-email accounts get 250 once.

Benchmark success. 72.0%, or 360 of 500, eighth of 16. All five attempts on 67 sites, partial on 11, nothing on 22.

Latency. A latency score of 16.32 seconds, sixth of 16.

Strengths. The most predictable bill here, because a protected page costs the same credit as an easy one. 96.0% against Cloudflare, the best of the seven other APIs, and 100% on jobs and grocery sites. An MCP server.

Weaknesses. 43.2% against DataDome, and nothing returned on 22 sites, among them Reddit, Trustpilot and Home Depot. Against Firecrawl, its closest hosted rival on price per page, Context.dev returned more attempts on 14 sites and Firecrawl on 24; they tied on 62.

Firecrawl: crawl-to-Markdown and the second-fastest latency score

Pricing. Standard is $99 a month billed monthly for 100,000 credits, or $0.99 per 1,000 pages; the page defaults to the annual view at $83. A scrape that returns no document is free, but a page that responds with a 403 or 404 is returned and costs a credit. Credits roll over on Scale and Enterprise; its billing docs limit that to annual plans. The free tier is 1,000 credits a month.

Benchmark success. 80.2%, or 401 of 500, fourth of 16. All five attempts on 75 sites, the most of any API after String. Nothing on 17, among them LinkedIn, Reddit, Instagram and TikTok.

Latency. A latency score of 9.11 seconds, second of 16.

Strengths. Crawl, map, search and scrape endpoints built for LLM input, an open-source core under AGPL-3.0, SOC 2 Type II, and an MCP server. 93.3% against Cloudflare, PerimeterX and Kasada, and 100% on jobs and ticketing sites.

Weaknesses. 64.4% on in-house protection systems and 25% on social media. A Firecrawl engineer wrote in the adapter pull request that it blocks some sites by policy. See our Firecrawl alternatives page for the full breakdown.

Bright Data: proxies, datasets and unblocking under one contract

Pricing. No plan near $100. Pay as you go is $1.50 per 1,000 successful requests, and the Scale tier is $499 a month for 383,000 requests. The free tier is 5,000 requests a month.

Benchmark success. 74.6%, or 373 of 500, sixth of 16. All five attempts on 63 sites, partial on 21, nothing on 16.

Latency. A latency score of 15.62 seconds, fourth of 16.

Strengths. The broadest catalogue here, with proxies, a browser API, SERP APIs and datasets, plus a free MCP server. 100% against PerimeterX and 100% on social media sites.

Weaknesses. 47.4% against DataDome and 25% on ticketing sites.

Oxylabs: bandwidth billing and 100% on grocery sites

Pricing. Web Unblocker is billed by bandwidth. Micro is 8GB for $75 a month, Starter 38GB for $325, and Advanced 88GB for $660. Code WU40 takes 40% off for six months. The pricing page lists a 1GB free trial; its docs describe a $5 welcome credit.

Benchmark success. 69.0%, or 345 of 500, ninth of 16. All five attempts on 62 sites, nothing on 24, among them Amazon, StubHub and Ticketmaster. The harness calls Web Unblocker without its JavaScript-render header; our Oxylabs page explains the drop from August.

Latency. A latency score of 15.86 seconds, fifth of 16.

Strengths. 100% on grocery sites, 95.6% on news and finance, and 86.7% against PerimeterX. A hosted MCP server at mcp.oxylabs.io; the old GitHub repo was archived on September 16, 2026.

Weaknesses. Bandwidth billing makes heavy pages expensive and forecasts hard. 33.3% against AWS WAF, 0% on ticketing, and nothing on the one Fastly site.

Apify: a marketplace of prebuilt scrapers

Pricing. Free includes $5 of monthly usage, Starter is $19 a month and Scale is $199, each as prepaid usage plus pay as you go. Compute runs $0.20 per unit on Starter and $0.16 on Scale, and residential proxies cost $8 per GB on Starter. Unused prepaid usage expires at the end of each billing cycle. Store Actors charge either per event or per usage, and each Actor's page says which. Web Fetch charges from $1.00 per 1,000 successful fetches plus a small fee per Actor start; a 404 it converts to Markdown counts as a fetch.

Benchmark success. 77.4%, or 387 of 500, fifth of 16, through the Web Fetch Actor. All five attempts on 69 sites, nothing on 16. Other Store Actors were not tested.

Latency. A latency score of 20.64 seconds, twelfth of 16.

Strengths. The Apify Store lists more than 68,000 Actors, so a scraper for a named site often already exists. It offers scheduling, storage, an MCP integration, and Crawlee, its open-source crawling library. 100% against Kasada and on real-estate sites.

Weaknesses. Each Actor is maintained by its own developer, so quality and upkeep vary, and a site redesign can break one. Costs depend on run time and retries, which makes a bill hard to predict before a test run. See our Apify alternatives page.

Oxylabs vs Apify: which is better for scraping?

They sell different things. Oxylabs Web Unblocker is a proxy endpoint that returns the page you ask for and bills by bandwidth. Apify is a platform of prebuilt scrapers, and its Web Fetch Actor does the same job as an unblocker: a URL in, the page out. On the September 2026 benchmark, Apify's Web Fetch Actor passed 77.4% of requests and Oxylabs passed 69.0%.

Site by site, Apify returned more attempts on 25 of the 100 sites and Oxylabs on 17. They tied on 58. Apify returned all five attempts on 69 sites and Oxylabs on 62. Oxylabs returned nothing on 24 sites and Apify on 16. Together they covered 81 sites in full.

Oxylabs led on Airbnb, Best Buy, Etsy, Lowe's, Kroger, Lululemon, Yelp, TikTok, Pinterest, Coupang and Reuters. Apify led on Amazon, Home Depot, Indeed, Glassdoor, ZipRecruiter, StubHub, SeatGeek, StockX, Realtor.com, Zoopla and Trustpilot. By protection, Apify led on Cloudflare (93.3% to 57.3%), AWS WAF (66.7% to 33.3%) and Akamai (75.0% to 67.0%). Oxylabs led on in-house systems (82.2% to 77.8%) and had the faster latency score, 15.86 seconds to 20.64.

Pick Oxylabs if you already buy its residential proxies and your list looks like the retail sites above. Pick Apify if you also want its Store scrapers and scheduling on the same account. Oxylabs Micro costs $75 for 8GB a month; Web Fetch costs from $1.00 per 1,000 successful fetches on top of an Apify plan.

Crawl4AI: open-source crawling for language models

Pricing. Free. Crawl4AI is open source under Apache-2.0 and runs on your own machines. Any LLM extraction you add bills through your own model provider.

Benchmark success. Not tested. Its results depend on the proxies and browser settings you supply.

Strengths. Built to turn pages into clean Markdown for RAG and agents, with a self-hosting guide and a large, active community on GitHub. No per-page fee at any volume.

Weaknesses. No proxy pool and no unblocking service are included, so a DataDome or Akamai site blocks it the way it blocks any headless browser. You also run and patch the infrastructure.

Scrapy: the Python framework for large crawls

Pricing. Free. Scrapy is open source under BSD-3-Clause.

Benchmark success. Not tested.

Strengths. A long-established Python crawling framework, with asynchronous request scheduling, auto-throttling, item pipelines and middlewares, as its overview describes. It is fast on sites that serve plain HTML and it scales to very large crawls.

Weaknesses. It does not run JavaScript by default. Its own documentation on dynamically loaded content points to parsing embedded data or adding a headless browser. It has no proxies or captcha handling, so protected sites need an API or proxy service behind it. Several APIs on this page, including Scrapfly, publish Scrapy integrations for that reason.

Scrapy vs a hosted scraping API: when to switch

Stay on Scrapy while your sites serve plain HTML and retries fix the blocks you see. Add an API when the same 403, 429 or 503 comes back after retries, when a 200 carries a challenge page instead of the content, when the data loads by JavaScript, or when proxies and captchas take more of the week than the data. You do not have to replace Scrapy. Keep the spider for crawling and send only the blocked pages to the API.

This spider does that. It is one file, it runs with scrapy runspider, and the middleware retries a blocked response once through String's fetch endpoint:

"""Keep Scrapy for the crawl. Send only the pages a site blocks to the String Web Access API.

    pip install scrapy
    export STRING_API_KEY=...
    scrapy runspider hybrid_spider.py -o pages.jsonl
"""
import json
import os

import scrapy
from scrapy.http import HtmlResponse

STRING_FETCH = "https://request.usestring.ai/v1/fetch"
BLOCK_STATUSES = {401, 403, 429, 503}


class StringFallbackMiddleware:
    """Retry a blocked response once through String; pass every other response through."""

    def process_response(self, request, response, spider):
        if "string_for" in request.meta:
            page = json.loads(response.text)
            return HtmlResponse(
                url=request.meta["string_for"],
                status=page["statusCode"],
                body=page["data"].encode(),
                encoding="utf-8",
                request=request.replace(url=request.meta["string_for"]),
            )
        if response.status in BLOCK_STATUSES:
            spider.logger.info("blocked (%s), retrying through String: %s", response.status, request.url)
            return scrapy.Request(
                STRING_FETCH,
                method="POST",
                body=json.dumps({"url": request.url}),
                headers={
                    "Authorization": f"Bearer {os.environ['STRING_API_KEY']}",
                    "Content-Type": "application/json",
                },
                meta={**request.meta, "string_for": request.url},
                callback=request.callback,
                dont_filter=True,
            )
        return response


class PagesSpider(scrapy.Spider):
    name = "pages"
    start_urls = [
        "https://books.toscrape.com/",  # open site: plain Scrapy, no API call
        "https://www.stockx.com/",  # protected site: Scrapy gets a 403, String returns the page
    ]
    custom_settings = {
        "DOWNLOADER_MIDDLEWARES": {__name__ + ".StringFallbackMiddleware": 585},
        "HTTPERROR_ALLOW_ALL": True,
        "RETRY_ENABLED": False,
        "DOWNLOAD_TIMEOUT": 90,
        "ROBOTSTXT_OBEY": False,
    }

    def parse(self, response):
        yield {
            "url": response.url,
            "status": response.status,
            "via_string": "string_for" in response.request.meta,
            "title": response.css("title::text").get(default="").strip(),
            "bytes": len(response.body),
        }

We ran it on October 5, 2026 with Scrapy 2.13.4. books.toscrape.com came back through plain Scrapy with no API call. stockx.com returned HTTP 403 to Scrapy, and the middleware sent it through String, which returned the page and its title. Only the blocked page used an API request, and String does not bill a request it cannot return, so the fallback costs nothing on a page it also fails. Scrapfly publishes a Scrapy integration for the same pattern. A 200 that carries a challenge page needs a content check as well as a status check; how to detect a bot block behind an HTTP 200 shows one.

Playwright: browser automation for pages that need a real browser

Pricing. Free. Playwright is Microsoft's open-source browser automation framework under Apache-2.0.

Benchmark success. Not tested.

Strengths. One API drives Chromium, Firefox and WebKit, as its getting-started guide shows, and it can intercept network requests to read the JSON a page loads.

Weaknesses. It is a testing tool first. A browser alone brings no residential IPs and no fingerprint management, so protected sites detect and block it. At scale you run and pay for every browser.

Octoparse: no-code scraping

Pricing. A free plan includes 10 tasks and 50,000 exported rows a month. Its pricing cards show Standard at $99 a month and Professional at $249; its FAQ quotes $69 and $249 billed annually, a 16% saving. Paid plans carry a 5-day money-back guarantee.

Benchmark success. Not tested.

Strengths. A desktop app where you click the data you want, with 500+ preset templates on Standard. Standard includes IP rotation, residential proxies, captcha solving and cloud runs.

Weaknesses. Standard allows 100 tasks; its card says up to 6 concurrent cloud processes and its FAQ says 3. Professional allows 250 tasks and 20. Click-built tasks break when a site changes its layout, and the app does not suit a pipeline that needs code review or version control.

The anti-bot cut

Success rates by protection vendor for the eight APIs, from the September 16 run. Site counts are in the column headers.

API DataDome (19) Akamai (20) Cloudflare (15) PerimeterX (9) AWS WAF (6) Kasada (3) Fastly (1) In-house (27)
String 92.6 98.0 100 100 100 93.3 100 96.3
Scrapfly 92.6 81.0 90.7 84.4 96.7 93.3 20.0 83.0
ScraperAPI 82.1 92.0 89.3 88.9 83.3 66.7 100 76.3
Firecrawl 77.9 84.0 93.3 93.3 83.3 93.3 100 64.4
Apify 65.3 75.0 93.3 88.9 66.7 100 0 77.8
Bright Data 47.4 79.0 88.0 100 86.7 66.7 80.0 72.6
Context.dev 43.2 70.0 96.0 66.7 66.7 66.7 100 83.0
Oxylabs 65.3 67.0 57.3 86.7 33.3 86.7 0 82.2

Among the seven other APIs, five lead at least one column: Scrapfly on DataDome and AWS WAF, ScraperAPI on Akamai, Context.dev on Cloudflare, Bright Data on PerimeterX, and Apify on Kasada. Scrapfly and Context.dev tie on in-house systems at 83.0%. That spread is why no one of them covers a mixed list.

Success rate by anti-bot vendor. Web Data Frontier Benchmark · 100 targets · 500 requests per provider · September 16, 2026. Current leaderboard

How many web scraping tools do you need? The stack count

Every page in this series carries one statistic computed from the run's per-site results. For a page about tools, the question is how many you have to buy.

Take the seven other APIs on this page: Scrapfly, ScraperAPI, Firecrawl, Apify, Bright Data, Context.dev and Oxylabs. Count the sites where at least one of them returned all five attempts, for the best possible combination at each size.

  • One tool: 75 of 100 sites, from Firecrawl.
  • Two tools: 93 of 100, from Scrapfly and Firecrawl.
  • Three tools: 98 of 100, adding Apify's Web Fetch Actor (Oxylabs in its place ties).
  • Four tools: 99 of 100, from Scrapfly, ScraperAPI, Firecrawl and Oxylabs.
  • Five to seven tools: still 99. No further API adds a site.

String alone returned all five attempts on 93 of 100 sites. Two other APIs together match it. Three pass it by five sites, and four by six. Each size adds less: 75, then 18 more, then 5, then 1.

The stack is real coverage. It also means several integrations, several sets of failure handling and several bills. At the plans nearest $100 a month, the two-tool stack costs $199 in plans before usage (Scrapfly Pro $100 and Firecrawl Standard $99), the three-tool stack $218 with Apify Starter or $274 with Oxylabs Micro, and the four-tool stack $423 (adding ScraperAPI Startup $149 and Oxylabs Micro $75). You also need routing code that knows which API to send each site to, and that code breaks when a site changes protection.

One site defeated every combination and String: temu.com. Six sites return in full through the stack and not through String: idealista.com, where String returned nothing, and coupang.com, footlocker.com, autotrader.com, allegro.pl and yelp.com, where String returned four of five.

Why pricing is hard to compare across web scraping tools

  1. The unit is different. String, Firecrawl and Context.dev price per request or credit. Oxylabs prices per gigabyte. Apify prices compute units and events. Octoparse prices tasks. The frameworks are free, and the proxies you buy for them are the real cost.
  2. A credit is not a page. One Context.dev credit is one scrape. One Scrapfly credit is a datacenter fetch, and the same page on residential proxies is 25. One ScraperAPI request to a Cloudflare, DataDome, PerimeterX or Akamai site is 10 credits.
  3. Failure billing differs. String and Bright Data bill only successes. Scrapfly bills failures when more than 30% of an hour's traffic fails, and 404s. Context.dev bills 404s. Firecrawl bills a 403 or 404 page that returns a document, and ScraperAPI bills 404s.
  4. Annual prices pose as monthly. Firecrawl shows $83 by default and bills $99 monthly. Octoparse's FAQ quotes $69, the annual rate.
  5. Unused credit often expires. String's credits roll over. Scrapfly's do not, Apify's prepaid usage expires monthly, and Firecrawl rolls over only on Scale. ScraperAPI's and Context.dev's do not roll over.

What $100 a month buys

  • String, $100: $0.20 per 1,000 standard fetches, $2.00 premium, $1.00 browser, $4.00 browser on premium. Failed requests free, credits roll over.
  • Scrapfly, $100: 1,000,000 credits, or $0.10 to $2.50 per 1,000 pages depending on setting. No rollover.
  • ScraperAPI: no $100 plan; Startup at $149 for 1,000,000 credits, 10 per protected-site request.
  • Firecrawl, $99: 100,000 credits, or $0.99 per 1,000 pages.
  • Context.dev, $99: 125,000 credits, or $0.79 per 1,000 pages.
  • Bright Data: no plan near $100; $1.50 per 1,000 successful requests pay as you go.
  • Oxylabs, $75: 8GB of Web Unblocker bandwidth.
  • Apify: no $100 plan; $19 or $199 of prepaid usage, Web Fetch from $1.00 per 1,000.
  • Octoparse: Standard at $99 on its pricing card.
  • Crawl4AI, Scrapy, Playwright: free software; your servers and proxies are the bill.

Web scraping MCP servers and what they cost

Every hosted API on this page publishes an MCP server, so an agent can call it as a tool. None of them charges extra for the MCP route. A call bills the same account and the same rate as the API.

Tool MCP server How MCP calls bill
String Hosted Same as the API; the docs say a call bills exactly as if you had pasted the key in. The product-help tool is free
Scrapfly Hosted Same credits as the API
ScraperAPI Hosted and local Same credits as the API
Firecrawl Hosted and local Keyed calls draw your credits; a keyless tier is free with a daily cap per IP address
Apify Hosted Each Actor run bills at that Actor's price; Web Fetch loads by default, from $1.00 per 1,000 fetches
Bright Data Hosted Same account; the 5,000 free monthly requests are the account's free tier, not an extra allowance
Context.dev Yes Same credits as the API
Oxylabs Hosted, mcp.oxylabs.io Uses Web Scraper API credentials, not Web Unblocker; no separate price published. The GitHub repo was archived Sep 16, 2026

So the cheapest MCP server is the cheapest API for your sites. At $100 a month, String bills $0.20 per 1,000 standard fetches and nothing for a failed one, which matters for an agent that retries.

Which web scraping tool should you pick?

  • Your sites are protected and the pages have to come back. String, at 97.0% on the benchmark and the fastest latency score.
  • You want one predictable number in the budget. Context.dev, one credit per page.
  • You crawl sites into Markdown for an LLM. Firecrawl for a hosted service, Crawl4AI if you want to run it yourself.
  • You want many APIs under one key. Scrapfly.
  • Your hard sites run Akamai. ScraperAPI, at 92.0% against Akamai.
  • Your list is grocery or news. Oxylabs, at 100% and 95.6%.
  • You need proxies, datasets and SERP data on one contract. Bright Data.
  • Someone already built a scraper for your site. Apify, after a test run to price it.
  • Your sites are open and your team writes Python. Scrapy. Writes TypeScript: Crawlee.
  • You already run Scrapy and some sites block it. Keep Scrapy and add the fallback middleware above.
  • Your pages need clicks or logins. Playwright, with an unblocking API behind it for protected sites.
  • You do not write code. Octoparse.
  • Your list is retail, marketplaces or grocery. Pick from our best e-commerce scraping APIs, which scores the same APIs on the 43 e-commerce sites alone. Overall rank is a poor guide there: the median provider's score swings 26.5 points between its best and worst kind of shop.

FAQ

What are the best web scraping tools for AI agents?

For an agent that must fetch live pages, a hosted API with an MCP server is the shortest path, because the agent calls one tool and gets the page back. String passed 97.0% of requests on the September 16, 2026 benchmark, with the fastest latency score, and ships a hosted MCP server that bills at the API rate. Scrapfly, ScraperAPI, Firecrawl, Apify, Bright Data, Context.dev and Oxylabs also publish MCP servers. Crawl4AI is the open-source option when you run your own infrastructure and your sites are not heavily protected.

How many web scraping tools do I need?

For a list of bot-protected sites, more than one API can help, but each one adds less. On our 100-site benchmark the best single API other than String returned all five attempts on 75 sites, the best pair on 93, the best three on 98 and the best four on 99. String alone returned 93. The two-tool stack costs $199 a month in plans at the $100 anchor, before usage and routing code.

What is the best no-code web scraping tool?

Octoparse is the established no-code option. Its free plan includes 10 tasks and 50,000 exported rows a month, and Standard costs $99 a month on its pricing card, with residential proxies and captcha solving included. We did not benchmark it. Its limits are task counts and concurrency, and click-built tasks need fixing when a site changes its layout.

Are open-source web scraping frameworks better than paid scraping APIs?

They answer different problems. Scrapy, Crawlee, Playwright and Crawl4AI are free and give full control, and they work well on sites that do not block automation. A paid API earns its price on protected sites, because it supplies the residential proxies, browser fingerprints and captcha handling a framework leaves to you. On our benchmark's 100 protected sites, the eight APIs on this page returned between 69.0% and 97.0% of requests.

Scrapy vs a hosted scraping API: when should I switch?

When retries stop fixing blocks, when pages need a browser, or when proxy and captcha work takes more time than the data. You can keep Scrapy for the crawl and send only blocked pages to an API; the section above has a working middleware.

Managed scraping service vs scraping API: which do I need?

A scraping API returns the pages you request, and your code decides what to extract. A managed service builds and runs the collection and delivers the data. Our managed scraping service vs scraping API page covers the choice, the SLA to ask for and the cost.

Which web scraping tools have a free tier?

String gives the first 5,000 standard requests free. Scrapfly and ScraperAPI give 1,000 credits, and ScraperAPI adds a 7-day, 5,000-credit trial. Firecrawl gives 1,000 credits a month, Context.dev 1,000 a month on a work email, and Bright Data 5,000 requests a month. Apify's free plan includes $5 of monthly usage and Octoparse has a free plan with 10 tasks. Scrapy, Crawlee, Playwright and Crawl4AI are free open-source software.

Can I verify these benchmark numbers myself?

Yes. The harness, the 100 sites, the pass rule and every adapter are open on GitHub, and it runs with your own API keys. Per-site results for all 16 providers are on the benchmark page, and our three-way comparison of ScraperAPI, ZenRows and ScrapingBee shows how to test a shortlist on your own list.

Sources

Checked October 5, 2026. Benchmark figures: September 16, 2026 run, official_results/benchmark-2026-09-16T01-03-47-074Z.json in the harness repository.

Benchmark data

  1. https://usestring.ai/benchmark
  2. https://usestring.ai/blog/web-data-frontier-benchmark-september-2026
  3. https://usestring.ai/blog/web-scraping-benchmark-problem
  4. https://github.com/usestring/web-data-frontier-benchmark
  5. https://github.com/usestring/web-data-frontier-benchmark/pull/6

Hosted APIs

  1. https://usestring.ai/pricing
  2. https://portal.usestring.ai/docs/get-started/pricing
  3. https://portal.usestring.ai/docs/mcp/overview
  4. https://portal.usestring.ai/docs/fetch/structured-extraction
  5. https://scrapfly.io/pricing
  6. https://scrapfly.io/docs/scrape-api/billing
  7. https://scrapfly.io/compliance
  8. https://scrapfly.io/products/mcp-cloud
  9. https://scrapfly.io/docs/scrape-api/anti-scraping-protection
  10. https://scrapfly.io/docs/sdk/scrapy
  11. https://www.scraperapi.com/pricing/
  12. https://docs.scraperapi.com/getting-started/quick-start/credits-and-requests-costs
  13. https://docs.scraperapi.com/integrations/llm-integrations/mcp-server
  14. https://www.context.dev/pricing
  15. https://docs.context.dev/install-mcp
  16. https://www.firecrawl.dev/pricing
  17. https://docs.firecrawl.dev/billing
  18. https://docs.firecrawl.dev/mcp-server
  19. https://github.com/firecrawl/firecrawl
  20. https://brightdata.com/pricing/web-unlocker
  21. https://brightdata.com/ai/mcp-server
  22. https://oxylabs.io/products/web-unblocker/pricing

Marketplace, frameworks and no-code

  1. https://apify.com/pricing
  2. https://apify.com/apify/web-fetch
  3. https://apify.com/store
  4. https://docs.apify.com/platform/actors/running/actors-in-store
  5. https://docs.apify.com/platform/integrations/mcp
  6. https://github.com/apify/crawlee
  7. https://github.com/apify/crawlee-python
  8. https://github.com/unclecode/crawl4ai
  9. https://docs.crawl4ai.com/
  10. https://github.com/scrapy/scrapy
  11. https://docs.scrapy.org/en/latest/intro/overview.html
  12. https://docs.scrapy.org/en/latest/topics/dynamic-content.html
  13. https://books.toscrape.com/
  14. https://github.com/microsoft/playwright
  15. https://playwright.dev/docs/intro
  16. https://playwright.dev/docs/network
  17. https://github.com/puppeteer/puppeteer
  18. https://github.com/SeleniumHQ/selenium
  19. https://www.crummy.com/software/BeautifulSoup/
  20. https://www.octoparse.com/pricing

Related String pages

  1. https://usestring.ai/answers/detect-bot-block-http-200
  2. https://usestring.ai/answers/managed-scraping-service-vs-scraping-api
  3. https://usestring.ai/comparisons/string-vs-scraperapi
Best Web Scraping APIs in 2026: 16 Tools ComparedUpdated September 22, 2026Web Scraping API Pricing in 2026: 16 Providers ComparedUpdated September 23, 2026Best E-commerce Scraping APIs in 2026: 8 APIs ComparedUpdated September 22, 2026
Get your API key →Explore the Web Access API
© 2026 StringEU and UK GDPR Article 27 representative — appointment verified by EuverifyBuilt in New York City 🗽 🍎