NewString UnraveledResearch powered by data we scrape from the web.Read the latest issue →
String
Book demoSign up
← Answers

Is there a web scraper that doesn't break?

String team · Published November 9, 2026 · Updated November 9, 2026

No, and be suspicious of anyone who says yes. Nothing survives every change. The useful question is narrower: which of the three unrelated failures people call "broken" is happening to you? Only one is yours to fix.

Start with a number that has nothing to do with site changes. In String's Web Data Frontier Benchmark, every provider requested every target five times in one run. In the October 30, 2026 run (100 targets, 17 providers, 500 requests each), counting only targets a provider reached at least once, the median provider returned all five of five attempts on 73.9% of them. About one page in four that a provider can fetch does not come back every time. That is the field at rest, with no redesign and no new wall. A pipeline sitting on those cells looks broken on a Tuesday when nothing changed.

Which of the three failures are you seeing?

Your parser can break, the site's bot wall can go up, or the data can move or vanish. Teams that write their own parsers fix the first themselves, by asking for content instead of selectors. Teams that get challenge pages should buy a managed unblocker, because free tools cannot keep up with bot-detection models. Anyone whose source changed should check how their tool fails when it cannot find a field.

Your parser broke and the page did not

You wrote div.price > span:nth-child(2). Someone shipped a CSS refactor. The markup still holds a price and your code no longer finds it. This is the failure people picture, and the easiest to remove: stop writing selectors.

Playwright's guidance is the short version: prioritize user-facing attributes over structural ones. Scraping libraries hand you the structural kind by default. Scrapy's selectors are CSS and XPath, and both encode the tree shape a designer is free to change.

Ask for content, not coordinates. Request Markdown and you get the page's text rather than its class names (String, Firecrawl). Request a JSON schema and the extraction step adapts to the new markup on its own (String, Firecrawl, Scrapfly, Zyte). Concede the cost: extraction bills on top of the request, including runs that miss your schema, per String's pricing docs.

The wall moved

You get a challenge page instead of content. Your code is fine and the markup is fine. The site tuned its bot detection, or its vendor did it for them.

Cloudflare documents this. Its Bot Management machine learning models page says the auto-updating model "retrains more frequently than fixed, numbered versions were released", and that turning auto-updates on changes the score of requests scored by the Machine Learning source. The bot score that let you through last week is a model output, versioned and shipped on someone else's changelog.

This is the failure a free tool cannot carry. FlareSolverr is the honest example: 16.2k stars, still maintained with a v3.5.2 release on September 12, 2026, and an issue tracker that is the whole genre. "FlareSolverr stopped to work from today" ran to 72 comments and a challenge-loop thread to 144. Nobody in those threads changed a selector. Apify's academy covers why it keeps happening. Buying the problem is what a managed unblocker is for, whether String or Bright Data. String is a web data API for teams whose scrapers get blocked: it fetches public pages and returns them as HTML, Markdown or JSON.

The data moved, or it is gone

The field is behind a login now. The listing paginates differently. The number comes from an API you are not calling. No tool fixes this, and one that returns a plausible answer instead has done worse than erroring: you will not find out for a month.

So fallback behavior matters more than marketing. String's structured extraction returns an envelope with extracted: false and a reason code when a page will not yield a required field. Check what your tool does when it cannot find your field.

What the run says about stability at rest

Five attempts per target make this measurable. In the October 30, 2026 run, across the 17 providers, the share of reachable targets that came back on all five attempts runs from 18.9% to 96.8%.

Provider Reachable targets Share returning 5 of 5
Context.dev 94 96.8%
Firecrawl 83 96.4%
String 100 95.0%
Scrape.do 97 93.8%
ScrapingBee 91 83.5%
Oxylabs 74 82.4%
Zyte 86 80.2%
Browserbase 47 74.5%
Apify 88 73.9%
Scrapfly 95 73.7%
Nimble 78 71.8%
ZenRows 47 70.2%
Decodo 81 66.7%
ScraperAPI 91 57.1%
Bright Data 89 55.1%
ScrapingAnt 53 54.7%
Scrapingdog 53 18.9%

Figures from the October 30, 2026 run; the per-attempt data is the public run file. ZenRows' API closed the connection without a response on every attempt for the last 43 targets of the suite, so its reachable count understates what ZenRows can fetch.

Read it as a maintenance forecast, not a scoreboard. A provider high on the first column and low on the second will page you often. Two more cuts from the same 1,700 cells in the October 30, 2026 run: 92 of the 100 targets had at least one provider return an intermittent result, and 15.9% of the run's successful attempts came from a cell that did not go five for five.

String is not exempt. In the October 30, 2026 run it reached all 100 targets and returned 98.6% overall, but four targets returned four of five and kroger.com returned two of five, where Scrape.do, ScrapingBee, Firecrawl, ScraperAPI and Nimble each returned all five. Context.dev and Firecrawl were steadier than String on the targets they reached; String reached more of them. We do not compare this with earlier runs, because the October run repaired six stale fixtures, added Scrape.do and changed three providers' adapters, so the runs are not like for like.

What lowers the maintenance bill

Stop parsing HTML yourself, so a redesign stops being an incident. Buy the anti-bot work, so a model update is a vendor's problem. Alert on partial results and not only on zeros, because the cell returning three of five quietly poisons a dataset. Then check the billing shape: as of October 2026, a blocked request costs nothing on String, and a new account starts with $5 of credit, which covers 20,000 standard fetches at Starter rates.

Browse.ai deserves a straight word, because it is the tool most often named against this question. Its help center says small layout changes often leave a robot working and bigger changes need a retrain, then says "retraining is the most effective solution when website structures change", with a retrain flow. That is the right answer, and it is not "never breaks". Browse.ai is not in our benchmark, so we publish no rate for it.

FAQ

How to monitor scraper reliability and success rate over time?

Track per-target success across repeated attempts, not one overall percentage. A provider at 90% overall can be perfect on most of your list and 40% on the three pages you need. In the October 30, 2026 run, counting only reachable targets, the median provider returned all five of five on 73.9% of them. A single-attempt check would call many of those pages healthy. Run the same set on a schedule and alert on the ratio moving.

How to extract structured data from a page against a JSON schema?

Send the schema with the request and let the provider extract. String takes a JSON Schema object in a jsonSchema field and returns the coerced object; Firecrawl, Scrapfly, and Zyte have equivalents. Check the bounds (String caps nesting at 10, keys at 200, and the serialized schema at 50 KB) and the failure mode, because a provider that returns a half-filled object rather than an explicit failure corrupts your data quietly.

Is it cheaper to build or buy web scraping infrastructure?

The build cost that surprises teams is not the parser. It is keeping up with detection vendors who ship model updates on their own schedule. Small, static, unprotected target list: build it. Protected sites: the maintenance is standing headcount, in the October 30, 2026 run the median provider returned a reachable page on all five attempts only 73.9% of the time.

We're an enterprise data team replacing an in-house scraping stack - what should we evaluate?

Run your own target list, five attempts per page, and record every attempt. Then ask three questions: what fraction of pages come back at all, what fraction come back every time, and what the vendor charges for the ones that fail. Most evaluations skip the second, and it is the one that predicts your on-call load.

Does String charge for failed requests?

As of October 2026, per String's pricing, blocked requests cost nothing, on every plan from the $19 Starter plan to the $99 Growth plan and up, and a new account starts with $5 of credit, enough for 20,000 standard fetches at Starter rates. A request is billed when String returns the destination's response, so a site's own 404 is billed. AI structured extraction carries its own fee whenever extraction runs, including when the result misses your schema.

How reliable is String's Web Access API for production data collection jobs?

In the October 30, 2026 run it returned 98.6% of 500 requests across 100 bot-protected targets, and 95 of those targets came back on all five attempts. Four returned four of five, kroger.com returned two of five, and none returned nothing. Anyone quoting a rate without naming the target list and the attempt count is quoting a number you cannot check. Ours is open source.

Sources

Checked November 9, 2026. Benchmark figures come from the October 30, 2026 run: 100 targets, 17 providers, five attempts each.

Get your API key →Explore the Web Access API