Block, error, or bug? Classifying failed requests

Adapted from my lightning talk at Prague Crawl 2026, hosted by Apify and Massive. Watch the talk.

When you request a page you start with a URL and want one answer back: did the server give me the page I asked for? If the fetch fails loudly, with a timeout or a 500, you notice and retry. If it fails quietly, you don’t. You think you have the product page, but you actually have a CAPTCHA, and that CAPTCHA moves down your pipeline as if it were data. It turns into bad rows, then gaps, then a customer asking why their numbers moved.

Status codes don’t tell you much

The obvious check is the HTTP status code, and it doesn’t work. Block pages come back as 200, 202, 429, 418, or whatever the site or its antibot vendor decided to send. Titles are unreliable and so are bodies. A person can tell a block page from the real thing at a glance, but nobody looks at a billion responses.

The scraping vendors have the same problem. Scraping APIs regularly return a 2xx for a page that isn’t the page you asked for. If that HTML goes into a training set, your model learns from “verify you are human.”

A benchmark

We run an open benchmark, web-data-frontier-benchmark: 100 targets, most behind Akamai, DataDome, Cloudflare, PerimeterX or similar, with 5 requests per target through each provider’s standard API. A request passes only if it comes back 2xx and the body contains a token that appears on the real page. The scorer does no block detection of its own, so the interesting failure is easy to count: the provider said success, but the page wasn’t there.

From the September 2026 run (500 requests per provider):

Across the 15 third-party providers, 438 of 5,279 successful-looking responses (about 1 in 12) weren’t the page. Everything else a provider got wrong, it at least reported as an error.

Some caveats. “Not the page” includes stale fixtures, where the site changed and our token went with it, so String’s 1% is a rough floor for that noise. We also build one of the providers being measured. Raw per-attempt results are in the repo, so check our work. Read it for the pattern, not the ranking: every provider’s success signal is wrong some of the time. That’s why we built our own.

There are more than two outcomes

“Success or failure” isn’t enough. There are at least five end states, and each one calls for a different response:

Each site owner picks their own format and rules, so this list will never be complete. It is still useful, because confusing any two of these leads to the wrong next step. If you retry a 404 you waste money. If you treat a block as a server error, you back off politely while the antibot keeps blocking you.

How we classify

This is a five-class classification problem, with one constraint that drives the design: it runs on the hot path, on every request, at billions of requests. That gives a budget of a few milliseconds per request and close to zero marginal cost.

So we built it as a waterfall:

1. Cheap features, on the hot path. Status code, response size, time to first byte, a DOM fingerprint, headers, language. These are engineered and deterministic, and all of them take milliseconds to compute.
2. A small classifier with priors, also on the hot path. We use a per-domain prior when we have one and a global model when we don’t.
3. A heavy model for hard cases, offline. A larger model labels the long tail asynchronously. Within hours those labels go back into training and produce new heuristics.

We scored candidate models on macro F1 across the five classes. Mistakes on block and partial responses counted extra, because those are the expensive ones.

Rule-based heuristics were the baseline: good on patterns we already knew, bad on anything new. Logistic regression was fast and easy to interpret, but missed too many non-linear interactions. A hybrid of the two had the best F1 within the latency budget, so that’s what we shipped. A small language model reading raw HTML also scored well, but it’s far too slow and expensive for the hot path, so it only runs offline as the hard-case labeller in step 3.

The feedback loop is the most important part. A clever one-shot model goes stale the first time a vendor changes its challenge page. A simple model that keeps retraining on fresh hard cases does not.

Takeaways

1. Detecting blocks is what makes everything downstream trustworthy. If you can’t tell a block from a page, nothing you do with the page can be trusted either.
2. Servers lie. Do your own research, build features, and don’t rely on the status code.
3. Use feedback. A continuous feedback loop beats a clever, complicated one-shot approach.

What this doesn’t cover yet

Plenty is still open: the long tail of sites outside the major antibot vendors, which needs more data; requests other than GET, and their 400s; responses that aren’t HTML; sites with more than one antibot layered together; and the whole question of what a correct parse should look like. That last one is a talk for 2027.

If you want to check our numbers, you can run the benchmark against the String Web Access API yourself. We’re also hiring.