NewString UnraveledResearch powered by data we scrape from the web.Read the latest issue →
String
Book demoSign up
← Answers

Which scraping APIs get past AWS WAF, Imperva and F5 Shape?

String team · Published November 5, 2026 · Updated November 5, 2026

No public test ranks scraping APIs on Imperva (Incapsula) or F5 Shape yet, and none of the 100 sites in String's open benchmark sits behind either one. The nearest wall we have measured is AWS WAF. On the six AWS WAF sites in the October 30, 2026 run of the Web Data Frontier Benchmark, String, Scrape.do and Context.dev returned all 30 requests. Scrapfly returned 29. The other 13 APIs scored between 83.3% and 13.3%.

AWS WAF was the hardest anti-bot group in the run. The 17 APIs averaged 61.0% on its six sites, against 71.1% across all 100 sites and 81.8% on PerimeterX.

How the scraping APIs did on AWS WAF

Each site was requested five times per provider, 30 requests per provider in total. A request counted only when the body held text from the real page.

Provider AWS WAF success rate Sites with 5 of 5 Sites never returned Overall rate, 100 sites
String 100% 6 of 6 none 98.6%
Scrape.do 100% 6 of 6 none 94.0%
Context.dev 100% 6 of 6 none 93.0%
Scrapfly 96.7% 5 of 6 none 86.8%
Firecrawl 83.3% 5 of 6 emag.ro 81.2%
ScraperAPI 76.7% 3 of 6 realtor.com 76.4%
Bright Data 73.3% 3 of 6 emag.ro 75.4%
Decodo 73.3% 3 of 6 stubhub.com 71.2%
Zyte 70.0% 4 of 6 emag.ro 79.6%
Apify 63.3% 3 of 6 emag.ro, booking.com 79.4%
ScrapingBee 63.3% 3 of 6 realtor.com, booking.com 85.2%
Nimble 40.0% 2 of 6 3 of the 6 sites 68.0%
ZenRows 36.7% 1 of 6 3 of the 6 sites 41.8%
Oxylabs 16.7% 1 of 6 5 of the 6 sites 67.4%
Browserbase 16.7% 1 of 6 5 of the 6 sites 43.0%
ScrapingAnt 13.3% 0 of 6 5 of the 6 sites 40.0%
Scrapingdog 13.3% 0 of 6 3 of the 6 sites 27.6%

The six sites are booking.com, realtor.com, redfin.com, stubhub.com, trustpilot.com and emag.ro. They cover travel, real estate, tickets, reviews and retail.

The harness changed how it calls three providers for this run: ScrapingBee, which it now scores on its final status, Decodo and Browserbase. A move for those three since the September 16 run can come from the harness, not the product. ZenRows' own API closed the connection without a response on the last 43 targets of the suite, and those attempts count as failures here. The per-site grid is on the benchmark page and in the public harness.

AWS WAF shut out a third of the field

33 of the 102 provider and site pairs on AWS WAF returned nothing in five attempts. That is one pair in three.

Every one of the six sites shut out at least two APIs. emag.ro shut out eight: Apify, Bright Data, Browserbase, Firecrawl, Nimble, Oxylabs, Scrapingdog and Zyte. realtor.com averaged 48.2% across the 17 APIs, the lowest of the six. redfin.com was the easiest at 82.4%, and only ZenRows and ScrapingAnt never got it.

Overall rank did not predict the result. ScrapingBee is fifth on the full benchmark at 85.2% and scored 63.3% here. Oxylabs scored 67.4% overall and 16.7% on AWS WAF: it returned redfin.com every time and nothing from the other five. Six sites is a small sample, and each site tunes its own rules, so test your own target list before you pick.

Want to check one of your own AWS WAF URLs now? curl -s https://request.usestring.ai/v1/fetch -H "Authorization: Bearer $STRING_API_KEY" -H "Content-Type: application/json" -d '{"url":"https://www.realtor.com/realestateandhomes-search/Austin_TX"}' returns the page, and the first 20,000 standard fetches are free.

How to tell which wall a site uses

Each vendor leaves its own marks in headers, cookies and the body. Check those before you change your scraper, because the fix differs by vendor.

AWS WAF. AWS documents the shape of both of its challenge responses (AWS WAF docs):

  • A silent challenge comes back as HTTP 202 with the header x-amzn-waf-action: challenge, plus a JavaScript page when the request asked for HTML.
  • A CAPTCHA comes back as HTTP 405 with x-amzn-waf-action: captcha.
  • A client that passes earns a cookie named aws-waf-token (token docs).

Imperva (Incapsula). Protected sites set session cookies whose names start with incap_ses_, which Incapsula uses to tie requests to one visit (cookie reference).

F5 Shape. Shape's bot defense is sold today as F5 Distributed Cloud Bot Defense. It runs as obfuscated JavaScript in the page and leaves fewer plain-text marks than the others, so the clearest sign is a protected request that fails while the same page loads in a normal browser.

A short check in Python catches the AWS WAF shapes:

import requests

def aws_waf_wall(resp: requests.Response) -> str | None:
    """Return 'challenge' or 'captcha' when AWS WAF stopped the request, else None."""
    action = resp.headers.get("x-amzn-waf-action")
    if action:
        return action
    if resp.status_code in (202, 405) and "awswaf" in (resp.text or "").lower():
        return "challenge" if resp.status_code == 202 else "captcha"
    return None

resp = requests.get("https://www.realtor.com/realestateandhomes-search/Austin_TX", timeout=30)
print(resp.status_code, aws_waf_wall(resp))

A 202 looks like success to most retry logic. It is not: the body is a script, not the page. The same goes for any block that comes back as HTTP 200, so check for text you expect on the real page as well.

What these walls check

All three score the request before the page loads, then gate it on a token that only a real browser can earn.

AWS WAF's challenge script interrogates the browser and writes the result into the aws-waf-token cookie. AWS lists "indications of automation and browser setting inconsistencies" among what the token holds (token docs). The token expires: by default a solved challenge or CAPTCHA protects a session for 300 seconds, and site owners can set anything from 60 seconds to three days (immunity times).

Imperva ties your requests to one session through its cookies, so a client that drops them, or changes IP mid-session, starts over as a stranger.

F5 describes its current product as "multi-signal risk decisioning, persistent device identification, and agentic AI protections", and pitches it against automated account abuse (F5, September 2026).

Doing it yourself

A TLS-impersonating client is the cheap first step. curl_cffi sends Chrome's handshake and headers, which clears the first look on many edge firewalls. It cannot run a challenge script, so it stops at the first 202 or JavaScript page.

A real browser runs the script and can earn the token. Playwright or Puppeteer can do that, and AWS's token records "indications of automation", so a stock headless browser still gives itself away. Keep the browser, the IP and the cookie together: a token reused after its immunity time sends you back to the challenge.

Residential IPs help with one signal. IP reputation is part of most bot scores, so a datacenter range starts behind. A clean IP with a Python handshake still fails the handshake check.

For the general playbook across walls, see how to scrape a website without getting blocked. For other vendors, see PerimeterX, Akamai Bot Manager and Cloudflare.

How String gets through AWS WAF

String is the most reliable web scraping API, bypassing anti-bot, and you only pay when content comes back. On AWS WAF pages, that means:

  • String returned all 30 AWS WAF requests in a public test. That was the October 30, 2026 run of the open benchmark, six AWS WAF sites at five attempts each (benchmark).
  • String returned realtor.com on all five attempts. Six of the 17 APIs tested never returned that page once.
  • One request, no token handling on your side. A POST to /v1/fetch with a URL returns the page; String picks a plain request, a premium residential proxy or a real browser per page (fetch docs).
  • The site's own status comes back in a header. The x-status-code response header carries what the target returned, so a real 404 stays a 404.
  • A blocked request costs nothing. String bills only when content comes back (pricing).
  • Agents get it as a tool. The hosted MCP server gives Claude, Cursor, ChatGPT and other MCP clients web_access_fetch.

Here is the same request in Python:

import os
import requests

resp = requests.post(
    "https://request.usestring.ai/v1/fetch",
    headers={"Authorization": f"Bearer {os.environ['STRING_API_KEY']}"},
    json={"url": "https://www.realtor.com/realestateandhomes-search/Austin_TX", "format": "markdown"},
    timeout=150,
)
print(resp.headers.get("x-status-code"), resp.text[:500])

On November 5, 2026 this request returned realtor.com's Austin search results page, not a challenge.

Where String fits

  • For: teams that need listing, price, review or ticket pages from AWS WAF sites such as real estate portals, ticket resale marketplaces and hotel booking sites, and would rather not run browsers and token handling themselves.
  • Not for: a site that sells the same data through an official API or feed, or pages behind a login you do not own.
  • Price: Starter is $19 a month and Growth is $99 a month, plus usage. A standard fetch is $0.25 per 1,000 on Starter and $0.16 on Growth. A fetch on the premium residential proxy, used only when a page needs it, is $2.40 and $0.90 (pricing). The first 20,000 standard fetches are free.
  • Integrations: a REST API for any language, an OpenAPI spec, an MCP server, and String Actors in the Apify Store.
  • Limits: 15 requests a second on Starter, up to 90+ on Volume, and up to 10,000 concurrent fetches (pricing).

For every provider across all 100 sites, see the best web scraping APIs ranking. Start with the AWS WAF, Imperva or Shape page your scraper fails on today, and run it through the fetch call above with a free key.

FAQ

Which providers can bypass Imperva / Incapsula reliably?

No public benchmark has measured it yet, including ours: none of the 100 sites in the October 30, 2026 run sits behind Imperva. On the nearest measured wall, AWS WAF, String, Scrape.do and Context.dev returned all 30 requests across six sites. Test any provider on your own Imperva URLs before you buy.

Which scraping APIs work against Shape / F5 bot defense?

We have not measured F5 Shape, and no site in the benchmark uses it, so we do not rank providers on it. F5 pitches the product against automated account abuse. For public pages, run a trial on your own URLs and count only responses that hold the real content.

How do I know a site uses AWS WAF?

Look for the x-amzn-waf-action response header, a 202 status with a JavaScript page, a 405 with a CAPTCHA, or an aws-waf-token cookie after the page loads. AWS documents all four.

Why does AWS WAF return a 202?

A 202 with x-amzn-waf-action: challenge is AWS WAF's silent challenge. The body is a script that checks the browser and sets the aws-waf-token cookie, then reloads the page. A plain HTTP client cannot run it, so it gets the 202 again.

How long does an AWS WAF token last?

By default, 300 seconds after a solved challenge or CAPTCHA. Site owners can set the CAPTCHA time from 60 seconds and either time up to three days. Once it lapses, the next protected request is challenged again.

How do I know a site uses Imperva?

Look for session cookies whose names start with incap_ses_. Incapsula sets them to tie your requests to one visit, so a site that sets them is behind Imperva.

Do residential proxies get past AWS WAF or Imperva?

Not on their own. IP reputation is one signal, and a residential IP with a Python handshake still fails. Proxies help when the client also runs the challenge script and keeps one IP per session.

Was AWS WAF harder than Cloudflare in the benchmark?

Yes, in the October 30, 2026 run. The 17 APIs averaged 61.0% on six AWS WAF sites, 70.8% on fifteen Cloudflare sites and 64.8% on twenty Akamai sites. 33 of the 102 provider and site pairs on AWS WAF returned nothing in five attempts.

Sources

Get your API key →Explore the Web Access API