How to scrape websites in n8n without getting blocked

Last updated: October 21, 2026. The n8n test on this page ran on October 21, 2026 in n8n 2.44.4, self-hosted on one US home connection. Benchmark figures come from the September 16, 2026 run of the Web Data Frontier Benchmark. The workflow and the curl command on this page ran on October 21, 2026, and the output shown is what they returned. We build String, the API in Step 6.

To scrape websites in n8n without getting blocked, check every response body, then send the pages that fail to an API built for blocked sites. Browser headers alone barely help. On October 21, 2026 we ran n8n's own HTTP Request node against 100 bot-protected pages, one request each. With n8n's defaults it got the real page on 42. With a full Chrome header set it got 45. The same node calling String's fetch endpoint got 90 by the same rule, and 94 after we checked four pages by hand. The steps below build that workflow: headers, a body check, pacing, retries, and a branch that routes blocked URLs to String.

This page is the n8n guide. For the reasons behind each technique in any language, see how to scrape a website without getting blocked.

On this page: TL;DR · Where String fits · What we measured · Why the HTTP Request node gets blocked · Step 1 to Step 7 · How to avoid getting blocked in n8n · How String handles blocked n8n requests · What it costs · FAQ · Sources

TL;DR

• n8n's HTTP Request node sends the User-Agent n8n unless you set one. n8n's own source code calls it the "legacy bare n8n User-Agent".
• A Chrome header set on the node added 3 real pages out of 100: 45 against 42.
• Cloudflare blocked the node on 15 of its 15 sites with Chrome headers, and on 14 of 15 with the defaults.
• 49 of the 100 pages came back on neither direct setup. String returned 43 of those 49 through the same HTTP Request node.
• Some blocks arrive as a 200. With Chrome headers, 6 responses were 2xx without the page. A Code node that checks the body catches them (Step 3).
• You do not need a community node. The built-in HTTP Request node calls String's REST API with a Header Auth credential.

Where String fits

String fits one job in an n8n workflow: the protected pages your HTTP Request node cannot get.

• For: n8n users on Cloud or self-hosted whose HTTP Request node returns 403s, "Just a moment..." pages or empty shells on Cloudflare, DataDome, Akamai or PerimeterX sites, and who want one extra node in place of a proxy pool and a browser container.
• Not for: sites with an RSS feed, a sitemap or a public API you can call, pages behind your own login, or unprotected sites where Steps 1 to 5 are enough and cost nothing.
• Price: you pay only when content comes back. The first 5,000 standard requests are free. Starter is $20 a month plus $0.30 per 1,000 standard fetches, $3.00 with premium proxies, $1.50 for a browser fetch and $6.00 for a browser fetch with premium proxies. Growth is $100 a month at $0.20, $2.00, $1.00 and $4.00 (pricing).
• Integrations: n8n's built-in HTTP Request node with nothing to install (String n8n guide), an MCP server for n8n AI agents and other clients, and String Actors on Apify Store.
• Limits: 60 requests a second per organization with a 3,600-request burst, and no cap on concurrent requests (pricing, rate limits). Some destinations need approval or KYC first and return a 403 that says which (fetch docs).

What we measured: n8n's HTTP Request node on 100 protected sites

We took the 100 target pages of the Web Data Frontier Benchmark and ran three n8n workflows against them, one request per page per workflow. Each workflow was a Manual Trigger, a Code node with the 100 URLs, and one HTTP Request node. We ran them headless with n8n execute, in chunks of about 12 pages, from one US home connection with no proxy. A page counts as real only when the status is 2xx and the body holds text from that page, such as the product name. That is the benchmark's own rule, so a challenge page served at 200 counts as a block.

The site's anti-bot vendor decides most of the result. The vendor labels come from the benchmark's target list.

Chrome headers helped most on DataDome sites (1 to 7) and cost two Akamai sites (8 to 6). Overall they won 9 pages, among them Walmart, Etsy, The New York Times and SeatGeek, and lost 6, among them American Airlines, eBay and Bloomberg.

Caveats: one request per page is a snapshot, a home IP usually scores better than n8n Cloud or a VPS, and the default run went out three hours before the other two. On October 20, 2026, plain Node.js fetch got 40 of the same 100 pages and fetch with Chrome headers 38.

String's 6 misses: two listings that no longer exist (CarGurus, MarketWatch), two sites that failed at the destination, and two calls past the node's 60-second timeout. The four hand-checked pages were three product and results pages whose wording had changed, and a Google results URL that String answers from its search endpoint.

Why the HTTP Request node gets blocked

The named failure here is a bot-shaped request. With no headers set, the node sends User-Agent: n8n and an Accept header that asks for JSON first; we read both back from httpbin.org through the node. n8n's source code calls it the "legacy bare n8n User-Agent", and N8N_ENFORCE_GLOBAL_USER_AGENT switches it to a browser-style string (n8n source).

The header is only the first layer. In our check against tls.peet.ws, the node's TLS handshake offered the same 52 cipher suites as Node's own fetch and no HTTP/2, with or without a Chrome User-Agent. Cloudflare scores TLS fingerprints as a bot signal (Cloudflare docs), and Chrome headers got 0 of its 15 sites.

n8n users report the same pattern. In February 2025 a user on r/n8n wrote that about 30% of the sites they fetched with the HTTP Request node returned the node's "Forbidden - perhaps check your credentials?" error (Reddit). In April 2026 a GitHub issue reported WAFs rejecting the bare n8n User-Agent with "Bad User-Agent" (n8n issue #28280). In May 2026 an n8n team member traced another user's 403 to CloudFront blocking the default User-Agent (n8n issue #30363). The same month, a user got Cloudflare's "Just a moment..." page from three blogs, and a top supporter on the n8n forum answered:

"on the bypass-cloudflare-natively question, no real way to do it in n8n." achamm, n8n Community, May 2026

Look for a feed first: one of those three blogs had a working /feed/ once a browser User-Agent was set. The RSS Read node and a sitemap cost nothing and rarely get blocked.

Step 1: build the workflow

Start with a trigger, a list of URLs and one HTTP Request node. Use a Manual Trigger while you build and a Schedule Trigger when it runs on its own. Give each URL a phrase that only the real page contains, such as a product name. Step 3 uses it.

Set these options on the HTTP Request node:

Here is the node with those options and the Step 2 headers. Copy it and paste it into the n8n editor with Cmd+V or Ctrl+V (n8n copy and paste):

{
  "nodes": [
    {
      "parameters": {
        "url": "={{ $json.url }}",
        "sendHeaders": true,
        "headerParameters": {
          "parameters": [
            {
              "name": "user-agent",
              "value": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/150.0.0.0 Safari/537.36"
            },
            {
              "name": "accept",
              "value": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8"
            },
            {
              "name": "accept-language",
              "value": "en-US,en;q=0.9"
            }
          ]
        },
        "options": {
          "batching": {
            "batch": {
              "batchSize": 4,
              "batchInterval": 1000
            }
          },
          "response": {
            "response": {
              "fullResponse": true,
              "neverError": true,
              "responseFormat": "text"
            }
          },
          "timeout": 30000
        }
      },
      "name": "Fetch direct",
      "type": "n8n-nodes-base.httpRequest",
      "typeVersion": 4.5,
      "position": [
        440,
        0
      ],
      "id": "a0000000-0000-4000-8000-000000000004",
      "onError": "continueRegularOutput"
    }
  ],
  "connections": {}
}

Never Error is the option that matters most. Without it, the first blocked URL fails the node and the rest of the batch never runs.

Step 2: set browser headers

Add a User-Agent, Accept and Accept-Language under Send Headers. This fixes sites that filter on the User-Agent alone. In our test it moved the node from 42 real pages to 45.

Our test sent 13 Chrome headers, including the sec-ch-ua and sec-fetch-* set. We left out zstd from accept-encoding, because the node does not decode it. If you self-host, you can also set N8N_ENFORCE_GLOBAL_USER_AGENT=true so every node sends a browser-style User-Agent (n8n source).

Headers will not get you past Cloudflare, DataDome or Akamai on most sites. Run this with a free key next: the URLs that still fail after Step 3 go to String in Step 6, and the first 5,000 standard requests cost nothing.

Step 3: check the response body

A 200 does not mean you got the page. Add a Code node after the HTTP Request node. It marks each item as blocked when the status is 403, 429 or 503, when the body holds a known challenge string, or when the page phrase is missing.

// Check the body, not only the status. A 200 can be a challenge page.
const WALLS = /Just a moment\.\.\.|cf-chl-|captcha-delivery\.com|px-captcha|Robot or human\?|Access Denied|Pardon Our Interruption/i;
return $input.all().map((item, i) => {
  const { url, mustContain } = $('URLs to scrape').itemMatching(i).json;
  const status = item.json.statusCode ?? 0;
  const body = typeof item.json.data === 'string' ? item.json.data : JSON.stringify(item.json.data ?? item.json.body ?? '');
  let blocked = '';
  if (!status) blocked = item.json.error ? `request failed: ${String(item.json.error.message ?? item.json.error).slice(0, 80)}` : 'no response';
  else if (status === 403 || status === 429 || status === 503) blocked = `HTTP ${status}`;
  else if (WALLS.test(body)) blocked = `challenge page ("${body.match(WALLS)[0]}")`;
  else if (!body.toLowerCase().includes(mustContain.toLowerCase())) blocked = `${status}, but the page text is missing`;
  return { json: { url, mustContain, status, blocked, body: blocked ? '' : body } };
});

Then add an If node with the condition {{ $json.blocked }} is not empty. The true branch holds the blocked pages.

In our test this check mattered. With Chrome headers, the node got 200 or 202 on 6 pages that were not the page: Zara and Autotrader served Akamai pages, Booking.com an AWS WAF challenge, Temu a verification shell with no prices, MarketWatch a search page, and Skyscanner an empty one. For more patterns, see how to detect a bot block that returns HTTP 200. If your If node sends URLs down the true branch today, Step 6 is where they go to String.

Step 4: pace your requests

By default, the HTTP Request node sends one request per input item, all at once. Two ways to slow it down:

• Use Batching in the node's options: Items per Batch and Batch Interval in milliseconds (HTTP Request docs). We used 4 items and 1,000 ms. The interval spaces out when each batch starts. It does not wait for the last batch to finish.
• Use Loop Over Items with a Wait node for real gaps between requests. Loop Over Items passes a few items at a time, and a Wait node set to After Time Interval holds the loop between them.

Pace each site on its own. At four requests a second, Hyatt, Expedia and Realtor.com answered 429 Too Many Requests on both direct setups. For backoff rules, see how to handle rate limiting when scraping.

Step 5: add retries and timeouts

Open the node's Settings tab and turn on Retry On Fail. Set Max Tries and Wait Between Tries (node settings). With Never Error on, a 403 is not a failure, so retries fire only on timeouts and connection errors. A 403 retried from the same IP rarely changes.

Two timeouts apply. The node's Timeout option covers only the wait for response headers (HTTP Request docs). A body that stops halfway is bounded by a separate setting, N8N_HTTP_RESPONSE_BODY_READ_TIMEOUT, which defaults to 300,000 ms (n8n source). A few of our executions stalled for minutes. Turn on Timeout Workflow in the workflow settings so one stuck page cannot hold the run (workflow settings).

Verizon and Yahoo Finance failed with "Header overflow": their response headers were too large for the node. If you self-host, Node's --max-http-header-size flag raises the limit (Node.js docs).

Step 6: route blocked URLs to String

Put the URLs the If node marked as blocked through String's fetch endpoint. First create a credential: Credentials > Add credential > Header Auth, Name Authorization, Value Bearer YOUR_API_KEY. The word Bearer and one space are part of the value (String n8n guide).

Then add a second HTTP Request node on the If node's true branch:

Leave Response Format on Autodetect. The page comes back as Markdown in $json.data, and the site's own status code in the x-status-code header (response formats). A page String cannot return comes back as a non-2xx status, so Retry On Fail covers it (String n8n guide).

To test one URL first, run this in a shell, or paste it into the HTTP Request node with Import cURL (HTTP Request docs).

curl https://request.usestring.ai/v1/fetch \
  -H "Authorization: Bearer $STRING_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"url": "https://www.homedepot.com/p/309752194", "format": "markdown"}'

On October 21, 2026 it returned status 200 with x-status-code: 200 and the Home Depot product page as 26 KB of Markdown. Keep the Content-Type header: without it the API answers 400.

Step 7: import the full workflow

This is the whole workflow: a Schedule Trigger every 6 hours and a Manual Trigger, the URL list, the direct request with Chrome headers and batching, the body check, the If node, and the String branch. Save it as a .json file and use Import from File, or copy it and paste it into the editor. Then pick your String credential in the String fetch node.

{
  "name": "Scrape without getting blocked (String fallback)",
  "nodes": [
    {
      "parameters": {
        "rule": {
          "interval": [
            {
              "field": "hours",
              "hoursInterval": 6
            }
          ]
        }
      },
      "name": "Every 6 hours",
      "type": "n8n-nodes-base.scheduleTrigger",
      "typeVersion": 1.2,
      "position": [
        0,
        -100
      ],
      "id": "a0000000-0000-4000-8000-000000000001"
    },
    {
      "parameters": {},
      "name": "Run now",
      "type": "n8n-nodes-base.manualTrigger",
      "typeVersion": 1,
      "position": [
        0,
        100
      ],
      "id": "a0000000-0000-4000-8000-000000000002"
    },
    {
      "parameters": {
        "jsCode": "// One item per page: the URL and a phrase only the real page contains.\nreturn [\n  { json: { url: 'https://books.toscrape.com/', mustContain: 'A Light in the Attic' } },\n  { json: { url: 'https://www.homedepot.com/p/Milwaukee-M18-FUEL-18-Volt-Lithium-Ion-Brushless-Cordless-Gen-II-18-Gauge-Brad-Nailer-Tool-Only-2746-20/309752194', mustContain: 'Brad Nailer' } },\n];"
      },
      "name": "URLs to scrape",
      "type": "n8n-nodes-base.code",
      "typeVersion": 2,
      "position": [
        220,
        0
      ],
      "id": "a0000000-0000-4000-8000-000000000003"
    },
    {
      "parameters": {
        "url": "={{ $json.url }}",
        "sendHeaders": true,
        "headerParameters": {
          "parameters": [
            {
              "name": "user-agent",
              "value": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/150.0.0.0 Safari/537.36"
            },
            {
              "name": "accept",
              "value": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8"
            },
            {
              "name": "accept-language",
              "value": "en-US,en;q=0.9"
            }
          ]
        },
        "options": {
          "batching": {
            "batch": {
              "batchSize": 4,
              "batchInterval": 1000
            }
          },
          "response": {
            "response": {
              "fullResponse": true,
              "neverError": true,
              "responseFormat": "text"
            }
          },
          "timeout": 30000
        }
      },
      "name": "Fetch direct",
      "type": "n8n-nodes-base.httpRequest",
      "typeVersion": 4.5,
      "position": [
        440,
        0
      ],
      "id": "a0000000-0000-4000-8000-000000000004",
      "onError": "continueRegularOutput"
    },
    {
      "parameters": {
        "jsCode": "// Check the body, not only the status. A 200 can be a challenge page.\nconst WALLS = /Just a moment\\.\\.\\.|cf-chl-|captcha-delivery\\.com|px-captcha|Robot or human\\?|Access Denied|Pardon Our Interruption/i;\nreturn $input.all().map((item, i) => {\n  const { url, mustContain } = $('URLs to scrape').itemMatching(i).json;\n  const status = item.json.statusCode ?? 0;\n  const body = typeof item.json.data === 'string' ? item.json.data : JSON.stringify(item.json.data ?? item.json.body ?? '');\n  let blocked = '';\n  if (!status) blocked = item.json.error ? `request failed: ${String(item.json.error.message ?? item.json.error).slice(0, 80)}` : 'no response';\n  else if (status === 403 || status === 429 || status === 503) blocked = `HTTP ${status}`;\n  else if (WALLS.test(body)) blocked = `challenge page (\"${body.match(WALLS)[0]}\")`;\n  else if (!body.toLowerCase().includes(mustContain.toLowerCase())) blocked = `${status}, but the page text is missing`;\n  return { json: { url, mustContain, status, blocked, body: blocked ? '' : body } };\n});"
      },
      "name": "Looks blocked?",
      "type": "n8n-nodes-base.code",
      "typeVersion": 2,
      "position": [
        660,
        0
      ],
      "id": "a0000000-0000-4000-8000-000000000005"
    },
    {
      "parameters": {
        "conditions": {
          "options": {
            "caseSensitive": true,
            "leftValue": "",
            "typeValidation": "loose",
            "version": 2
          },
          "conditions": [
            {
              "id": "c1",
              "leftValue": "={{ $json.blocked }}",
              "rightValue": "",
              "operator": {
                "type": "string",
                "operation": "notEmpty",
                "singleValue": true
              }
            }
          ],
          "combinator": "and"
        },
        "options": {}
      },
      "name": "Blocked?",
      "type": "n8n-nodes-base.if",
      "typeVersion": 2.2,
      "position": [
        880,
        0
      ],
      "id": "a0000000-0000-4000-8000-000000000006"
    },
    {
      "parameters": {
        "method": "POST",
        "url": "https://request.usestring.ai/v1/fetch",
        "authentication": "genericCredentialType",
        "genericAuthType": "httpHeaderAuth",
        "sendBody": true,
        "specifyBody": "json",
        "jsonBody": "={{ JSON.stringify({ url: $json.url, format: \"markdown\" }) }}",
        "options": {
          "response": {
            "response": {
              "fullResponse": true
            }
          },
          "timeout": 90000
        }
      },
      "name": "String fetch",
      "type": "n8n-nodes-base.httpRequest",
      "typeVersion": 4.5,
      "position": [
        1100,
        -100
      ],
      "id": "a0000000-0000-4000-8000-000000000007",
      "retryOnFail": true,
      "maxTries": 3,
      "waitBetweenTries": 2000,
      "credentials": {
        "httpHeaderAuth": {
          "name": "String API key"
        }
      }
    },
    {
      "parameters": {
        "jsCode": "// Pages that came back through String. The site's own status is in the x-status-code header.\nreturn $input.all().map((item, i) => {\n  const { url, mustContain, blocked: why } = $('Looks blocked?').itemMatching(i).json;\n  const text = String(item.json.data ?? '');\n  return { json: {\n    url, via: `String (direct: ${why})`, status: Number(item.json.headers?.['x-status-code'] ?? 0),\n    billed: item.json.headers?.['x-billed-request-type'], chars: text.length,\n    realPage: text.toLowerCase().includes(mustContain.toLowerCase()),\n  } };\n});"
      },
      "name": "Via String",
      "type": "n8n-nodes-base.code",
      "typeVersion": 2,
      "position": [
        1320,
        -100
      ],
      "id": "a0000000-0000-4000-8000-000000000008"
    },
    {
      "parameters": {
        "jsCode": "// Pages the direct request got.\nreturn $input.all().map((item) => ({ json: {\n  url: item.json.url, via: 'direct', status: item.json.status, chars: item.json.body.length,\n  realPage: item.json.body.toLowerCase().includes(item.json.mustContain.toLowerCase()),\n} }));"
      },
      "name": "Direct page",
      "type": "n8n-nodes-base.code",
      "typeVersion": 2,
      "position": [
        1100,
        100
      ],
      "id": "a0000000-0000-4000-8000-000000000009"
    }
  ],
  "connections": {
    "Every 6 hours": {
      "main": [
        [
          {
            "node": "URLs to scrape",
            "type": "main",
            "index": 0
          }
        ]
      ]
    },
    "Run now": {
      "main": [
        [
          {
            "node": "URLs to scrape",
            "type": "main",
            "index": 0
          }
        ]
      ]
    },
    "URLs to scrape": {
      "main": [
        [
          {
            "node": "Fetch direct",
            "type": "main",
            "index": 0
          }
        ]
      ]
    },
    "Fetch direct": {
      "main": [
        [
          {
            "node": "Looks blocked?",
            "type": "main",
            "index": 0
          }
        ]
      ]
    },
    "Looks blocked?": {
      "main": [
        [
          {
            "node": "Blocked?",
            "type": "main",
            "index": 0
          }
        ]
      ]
    },
    "Blocked?": {
      "main": [
        [
          {
            "node": "String fetch",
            "type": "main",
            "index": 0
          }
        ],
        [
          {
            "node": "Direct page",
            "type": "main",
            "index": 0
          }
        ]
      ]
    },
    "String fetch": {
      "main": [
        [
          {
            "node": "Via String",
            "type": "main",
            "index": 0
          }
        ]
      ]
    }
  },
  "settings": {
    "executionOrder": "v1",
    "executionTimeout": 600
  }
}

We ran this exact workflow in n8n 2.44.4 on October 21, 2026. The Direct page and Via String nodes returned:

{"url": "https://www.homedepot.com/p/Milwaukee-M18-FUEL-18-Volt-Lithium-Ion-Brushless-Cordless-Gen-II-18-Gauge-Brad-Nailer-Tool-Only-2746-20/309752194", "via": "String (direct: HTTP 403)", "status": 200, "billed": "request_premium", "chars": 27129, "realPage": true}
{"url": "https://books.toscrape.com/", "via": "direct", "status": 200, "chars": 51274, "realPage": true}

Books to Scrape came back on the direct request. Home Depot returned 403 to the direct request, and the String branch returned the product page.

How to avoid getting blocked in n8n

The checklist, in the order that paid off in our test:

• Use an RSS feed, sitemap or API when the site has one.
• Turn on Never Error and Include Response Headers and Status (Step 1).
• Set a browser User-Agent. It added 3 pages of 100 in our test (Step 2).
• Check every body for page text (Step 3).
• Pace requests per site (Step 4).
• Use Retry On Fail for timeouts and Timeout Workflow for stalls (Step 5).
• Use the node's Proxy option for a residential IP. It changes your IP and leaves the TLS handshake as it is (HTTP Request docs).
• Send what still fails to an API (Step 6).

For the reasons behind each technique, see the language-neutral guide. For a status-code walkthrough, see how to fix 403 Forbidden when scraping. For Cloudflare in particular, see how to bypass Cloudflare.

How String handles blocked n8n requests

String is the most reliable web scraping API, bypassing anti-bot, and you only pay when content comes back.

• One HTTP Request node calls String, with nothing to install, on n8n Cloud or self-hosted. The key lives in a Header Auth credential (String n8n guide).
• String returned 90 of 100 protected pages by the marker rule, 94 after a hand check, through n8n's HTTP Request node on October 21, 2026. The best direct setup in the same test, the node with Chrome headers, returned 45.
• String returned 43 of the 49 pages that neither direct setup got on October 21, 2026. In the September 16, 2026 benchmark run, String returned 43 of those 49 on all five attempts.
• String returned all 15 Cloudflare sites in the October 21, 2026 n8n test. The node with Chrome headers returned none of them.
• String had the highest success rate of 16 scraping APIs in the September 16, 2026 benchmark run: 97.0% across 100 sites. The field ran from 36.4% to 97.0% (benchmark).
• The x-billed-request-type response header names the fetch type each call used. On the 94 real pages, 54 calls ran as standard requests, 36 with premium proxies, 3 in a browser and 1 as a search (fetch docs).

What it costs

Our 100-page test made 95 billed fetch calls through String and one Google search. At Growth rates the fetch calls cost about $0.09 in total: 54 standard requests at $0.20 per 1,000, 38 premium-proxy requests at $2.00, 2 browser requests at $1.00 and 1 browser request with premium proxies at $4.00. At Starter rates the same calls cost about $0.14. The 54 standard calls fit inside the 5,000 free standard requests a new account gets (pricing). A destination 404 that comes back through String is billed, so the dead CarGurus listing counts (billing docs).

The direct branch costs nothing per request, so the String branch handles only the blocked items. A "format": "markdown" body also removes the HTML-cleaning nodes you would otherwise add after the fetch (String n8n guide).

Start with the URLs your workflow marks as blocked today. Import the Step 7 workflow, add a free key, and you will see your block rate on the first run.

FAQ

How do I scrape websites in n8n without getting blocked?

Turn on Never Error, set browser headers, check each body in a Code node, pace requests, and route blocked URLs to a fetch API. On October 21, 2026, the node got 42 of 100 protected pages with its defaults, 45 with Chrome headers, and 90 through String's fetch API by the same rule (94 after a hand check of four pages).

Why does the n8n HTTP Request node say "Forbidden - perhaps check your credentials?"

n8n shows that message for every 403. On a public page it means the site blocked the request, usually because of the default User-Agent: n8n or a TLS handshake that anti-bot systems score as a bot. Credentials are not the problem when the page needs none.

Does setting a User-Agent in the HTTP Request node stop 403 errors?

On simple filters, yes. On protected sites, a Chrome header set raised our count from 42 to 45 of 100 pages and got none of the 15 Cloudflare sites.

How do you scrape Cloudflare-protected websites in n8n?

Check for an RSS feed first. The HTTP Request node alone got 1 of 15 Cloudflare sites in our test with its defaults and 0 with Chrome headers. String, called from the same node, returned all 15. A self-hosted browser container is the other route.

How do I add a delay between requests in n8n?

Use the HTTP Request node's Batching option, which spaces out the start of each batch, or put a Wait node inside a Loop Over Items loop for a real gap between requests.

Can I use a proxy with the n8n HTTP Request node?

Yes. The node's Proxy option takes precedence over the HTTP_PROXY, HTTPS_PROXY and ALL_PROXY environment variables. A residential proxy changes your IP. Your headers and TLS handshake stay the same.

How do I tell if a 200 response in n8n is a block page?

Test the body in a Code node for text only the real page carries and for challenge strings such as "Just a moment...". In our test, 6 responses with Chrome headers were 2xx without the page.

How do I scrape JavaScript-heavy websites in n8n?

The HTTP Request node does not run JavaScript, so it gets the shell. String's fetch endpoint picks a browser when a site needs one, and executeJS: true forces it (JavaScript rendering). In our test, 3 of String's 94 real pages ran in a browser.

Sources

• n8n block test, October 21, 2026: one request per page per setup to the 100 Web Data Frontier Benchmark target pages, run in n8n 2.44.4 (HTTP Request node version 4.5) on Node.js 24.16.0.
• Web Data Frontier Benchmark, September 16, 2026 run: 16 APIs, 100 sites, five attempts per site. Harness and raw results. In that run, Scrapfly, Firecrawl, ScraperAPI, Zyte, ScrapingAnt, ScrapingBee, Scrapingdog and Nimble ran below their current anti-bot setting.
• n8n docs: HTTP Request node
• n8n docs: node settings (Retry On Fail, On Error)
• n8n docs: workflow settings (Timeout Workflow)
• n8n docs: Schedule Trigger, Manual Trigger, Code, If, Loop Over Items, Wait, RSS Read
• n8n docs: export, import, copy and paste
• n8n source: HTTP request config (User-Agent and body read timeout)
• n8n issue #28280: bare "n8n" User-Agent blocked by WAFs
• n8n issue #30363: 403 from CloudFront on the HTTP Request node
• n8n Community: How do you scrape Cloudflare-protected websites in n8n?
• Reddit r/n8n: 30% of websites block my HTTP requests in n8n
• Cloudflare: JA3/JA4 fingerprint
• tls.peet.ws fingerprint echo
• Node.js CLI: --max-http-header-size
• String n8n guide, fetch overview, response formats, JavaScript rendering, rate limits, billing, MCP server
• String pricing, read October 21, 2026