How to scrape websites in Node.js without getting blocked

Last updated: October 26, 2026. The Node.js test on this page ran on October 20, 2026 from one US home connection. Benchmark figures come from the September 16, 2026 run of the Web Data Frontier Benchmark. Every code block ran on Node 24.16.0 on October 20, 2026, and the output shown is what it printed. We build String, the API in Step 8.

To scrape a website in Node.js without getting blocked, fix the TLS handshake before the headers. Node's built-in fetch does not send Chrome's TLS handshake, so a Chrome User-Agent on top of it is a mismatch that anti-bot systems score. On October 20, 2026 we sent one request to each of 100 bot-protected pages. Plain fetch got the real page on 40. Adding a full Chrome header set dropped it to 38. got-scraping got 50. String's fetch endpoint got 86 by the same rule, and 90 once we checked four real pages whose marker text had changed. Then keep one cookie jar per session, cap concurrency, use residential IPs, render JavaScript pages in Playwright, and check every body for the content you expect.

This page is the Node.js tutorial. For the language-neutral guide, see how to scrape a website without getting blocked.

On this page: TL;DR · Where String fits · What we measured · Why Chrome headers made Node worse · Step 1 to Step 8 · How to avoid getting blocked in Node.js · How String handles blocked Node.js requests · What it costs · FAQ · Sources

TL;DR

• Node's fetch (undici) sends a TLS handshake that offers HTTP/1.1 only and 52 cipher suites. A Chrome-impersonating client offers HTTP/2 and 15. Servers see that before they read a header.
• A Chrome header set on plain fetch lost 6 pages that the default headers got and won 4 others. Net result: 38 real pages against 40.
• got-scraping 4.2.2 was the best plain Node client in our test: 50 of 100 pages.
• Cloudflare blocked plain fetch on all 15 of its sites, with or without Chrome headers.
• 46 of the 100 pages came back on none of the four Node clients. String's fetch endpoint returned 39 of those 46.
• Many blocks arrive as a 200. On the got-scraping rung, 9 responses were 2xx without the page. Check the body.

Where String fits

String fits one job on this page: the protected pages your Node scraper cannot get.

• For: Node.js and TypeScript teams whose fetch, axios or got scrapers hit 403s, challenge pages or empty shells on Cloudflare, DataDome, Akamai or PerimeterX sites, and who want one HTTP call in place of a proxy pool and a browser fleet.
• Not for: pages behind your own login, sites that publish an API you can use, or low-volume jobs on unprotected sites, where Steps 1 to 5 are enough and cost nothing.
• Price: you pay only when content comes back. The first 5,000 standard requests are free. Starter is $20 a month plus $0.30 per 1,000 standard fetches, $3.00 with premium proxies, $1.50 for a browser fetch and $6.00 for a browser fetch with premium proxies. Growth is $100 a month at $0.20, $2.00, $1.00 and $4.00 (pricing).
• Integrations: a REST endpoint you call with Node's own fetch (Step 8), an MCP server for agents, guides for the Vercel AI SDK and LangChain, n8n and Zapier, and String Actors on Apify Store.
• Limits: 60 requests a second per organization with a 3,600-request burst, and no cap on concurrent requests (pricing, rate limits). Some destinations need approval or KYC first and return a 403 that says which (fetch docs).

What we measured: four Node clients on 100 protected sites

We took the 100 target pages of the Web Data Frontier Benchmark and sent one GET to each page per client, four at a time, from one US home connection with no proxy. A page counts as real only when the status is 2xx and the body holds text from that page, such as the product name. That is the benchmark's own rule, so a challenge page served at 200 counts as a block.

The site's anti-bot vendor decides most of the result. The vendor labels come from the benchmark's target list.

Every Node client got 7 of Akamai's 20 sites, and 11 Akamai sites came back on none of them. Walmart's PerimeterX page answered every Node client with a 200 titled "Robot or human?". For how the scraping APIs fare on that wall, see scraping APIs on PerimeterX (HUMAN).

Three caveats. One request per page is a snapshot, so a page can flip on a second try. A home IP usually scores better than a cloud IP, so expect worse numbers from a server. And on October 7, 2026 we ran the same 100 pages through Python clients: default requests got 28, Chrome headers 48, and curl_cffi with Chrome impersonation 61. Python's default headers did worse than Node's, Chrome headers helped Python and not Node, and Python's best client got 11 more pages than Node's best. The two runs are 13 days apart, so sites may have changed in between (Python version of this test).

The 10 pages that did not count for String: two listings that no longer exist on the site (a CarGurus car that returns 404, a MarketWatch ticker that now redirects to search), a Google results URL that String routes to its search endpoint instead, three sites that failed at the destination, two calls that ran past our 60-second client timeout, and two responses whose headers were over Node's 16 KB header limit. Those last two came back in full on a re-run with node --max-http-header-size=131072.

Why Chrome headers made Node fetch worse

The named failure here is a fingerprint mismatch. A Chrome User-Agent says "I am Chrome". The TLS handshake under it says "I am Node". Cloudflare documents JA3 and JA4 TLS fingerprints as bot signals it scores (Cloudflare docs). Our language-neutral guide covers how DataDome, Akamai, Kasada and HUMAN use the same signal.

In our test the Chrome header set cost plain fetch 6 pages: Zillow, American Airlines, Ashley Furniture, Ralph Lauren, Skyscanner and Yahoo Finance. Yahoo Finance failed for a different reason: its response headers went over undici's size limit. The same header set won 4 pages: Amazon, Barron's, The Wall Street Journal and Redfin.

Node developers have reported this pattern for years. In October 2025 a developer on r/webscraping wrote that a Python requests script got a 200 while axios, Bun's fetch and got all got a 403, even with headers copied from the browser's network tab (Reddit). In 2023 a user on the undici repo showed node:https getting a 200 and undici a 403 on a Cloudflare site with the same headers (undici discussion #2249). On Stack Overflow, an answer about a Cloudflare-protected host put it plainly:

"The server doesn't accept requests made by fetch, but the do accept requests from curl." Δ O, Stack Overflow, November 2020

The steps below fix the layers in order: body check, fingerprint, client, session, pace, IP, browser, and a fallback.

Step 1: fetch the page and check the body

Start with the check, because a 200 does not mean you got the page. Pick a string that only the real page carries, such as a product name, and test for it.

// Step 1: fetch a page and check the body, not only the status.
const WALLS = /Just a moment\.\.\.|cf-chl-|captcha-delivery\.com|px-captcha|Robot or human\?|Access Denied|Pardon Our Interruption/i;

export function looksBlocked(status, html, mustContain) {
  if (status === 403 || status === 429 || status === 503) return `HTTP ${status}`;
  const wall = html.match(WALLS);
  if (wall) return `challenge page ("${wall[0]}")`;
  if (!html.toLowerCase().includes(mustContain.toLowerCase())) return "200, but the page text is missing";
  return null; // looks like the real page
}

const pages = [
  ["https://books.toscrape.com/", "A Light in the Attic"],
  ["https://stackoverflow.com/questions/11227809/why-is-processing-a-sorted-array-faster-than-processing-an-unsorted-array", "sorted array"],
  ["https://www.walmart.com/ip/Nike-Men-s-Short-Sleeve-Just-Do-It-Swoosh-Graphic-Active-T-Shirt-Navy-L/2501897203", "Just Do It Swoosh"],
];

for (const [url, mustContain] of pages) {
  const res = await fetch(url, { signal: AbortSignal.timeout(30_000) });
  const html = await res.text();
  console.log(res.status, new URL(url).hostname, looksBlocked(res.status, html, mustContain) ?? "real page");
}

Output on October 20, 2026:

200 books.toscrape.com real page
403 stackoverflow.com HTTP 403
200 www.walmart.com challenge page ("Robot or human?")

Walmart is the case that breaks naive scrapers: status 200, 15 KB, and no product. If your logs show lines like these today, Step 8 is where you test String on them with a free key.

Step 2: check your TLS fingerprint

tls.peet.ws echoes back the JA4 fingerprint of the client that connected. Compare Node's fetch with impit, a client that copies Chrome's handshake.

// Step 2: see what your TLS handshake says about you.
// tls.peet.ws echoes back the JA4 fingerprint of the client that connected.
import { Impit } from "impit";

const UA = "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/150.0.0.0 Safari/537.36";

const node = await (await fetch("https://tls.peet.ws/api/clean", { headers: { "user-agent": UA } })).json();
const chrome = await (await new Impit({ browser: "chrome" }).fetch("https://tls.peet.ws/api/clean")).json();

console.log("Node fetch + Chrome UA:", node.ja4);
console.log("impit (chrome):        ", chrome.ja4);

Output:

Node fetch + Chrome UA: t13d5212h1_b262b3658495_8e6e362c5eac
impit (chrome):         t13d1516h2_8daaf6152771_02713d6af862

Read the first part of each JA4 (FoxIO JA4 spec). d5212h1 means 52 cipher suites, 12 extensions and HTTP/1.1. d1516h2 means 15 cipher suites, 16 extensions and HTTP/2. The User-Agent header did not change Node's fingerprint at all.

Step 3: use got-scraping for browser-consistent requests

got-scraping is Apify's wrapper around got. It generates a consistent Chrome header set with Apify's header-generator, negotiates HTTP/2 the way browsers do, and ships a TLS configuration tuned to look closer to a browser's (README). In our test it was the best plain Node client.

// Step 3: send a browser-consistent request with got-scraping.
import { gotScraping } from "got-scraping";

const url = "https://www.etsy.com/listing/4409872205/linen-sheer-cafe-curtains-farmhouse";

const plain = await fetch(url, { signal: AbortSignal.timeout(30_000) });
console.log("node fetch:  ", plain.status);

const res = await gotScraping({
  url,
  headerGeneratorOptions: { browsers: [{ name: "chrome" }], devices: ["desktop"], operatingSystems: ["macos"] },
  timeout: { request: 30_000 },
  throwHttpErrors: false,
});
console.log("got-scraping:", res.statusCode, res.body.includes("Linen Sheer Cafe Curtains") ? "real page" : "not the page");

Output:

node fetch:   403
got-scraping: 200 real page

got-scraping got 9 pages that impit did not, among them Etsy, Indeed, SeatGeek, The New York Times and Congress.gov. impit got 2 pages that got-scraping did not: Lululemon and Alibaba. A closer TLS match alone did not win in our test. We ran impit with its default headers and added none.

Run this with a free key next: put the URLs that still fail after Step 3 through Step 8. The first 5,000 standard requests cost nothing.

Step 4: keep one cookie jar per session

Anti-bot systems set cookies on the first response and expect them back. A client that drops them looks new on every request. Give each session its own tough-cookie jar and pass it to every request in that session.

// Step 4: keep one cookie jar per session, and reuse it for every request in that session.
import { gotScraping } from "got-scraping";
import { CookieJar } from "tough-cookie";

function newSession() {
  const cookieJar = new CookieJar();
  return (url) => gotScraping({ url, cookieJar, timeout: { request: 30_000 }, throwHttpErrors: false });
}

const session = newSession();
await session("https://httpbin.org/cookies/set?session_id=abc123"); // the site sets a cookie
const res = await session("https://httpbin.org/cookies"); // the next request sends it back
console.log(res.statusCode, res.body.replace(/\s+/g, " "));

Output:

200 { "cookies": { "session_id": "abc123" } }

Pair one jar with one proxy IP (Step 6). A cookie that moves between IPs is a signal of its own.

Step 5: cap concurrency and back off

Node makes it easy to fire 500 requests at once, and that burst is a signal too. p-limit caps how many run at a time. Add jitter, and on a 429 or 503 wait for Retry-After or back off exponentially.

// Step 5: cap concurrency, add jitter, and back off on 429 and 503.
import pLimit from "p-limit";
import { setTimeout as sleep } from "node:timers/promises";
import { gotScraping } from "got-scraping";

const limit = pLimit(2); // two requests at a time to one site
const jitter = () => 800 + Math.random() * 1700; // 0.8 s to 2.5 s

async function politeGet(url, tries = 4) {
  for (let attempt = 1; attempt <= tries; attempt++) {
    await sleep(jitter());
    const res = await gotScraping({ url, timeout: { request: 30_000 }, throwHttpErrors: false });
    if (res.statusCode !== 429 && res.statusCode !== 503) return res;
    const wait = Number(res.headers["retry-after"]) * 1000 || 2 ** attempt * 1000;
    console.log(`${res.statusCode} on ${url}, waiting ${wait} ms`);
    await sleep(wait);
  }
  throw new Error(`gave up on ${url}`);
}

const urls = [1, 2, 3, 4, 5, 6].map((n) => `https://books.toscrape.com/catalogue/page-${n}.html`);
const t0 = Date.now();
const results = await Promise.all(urls.map((u) => limit(() => politeGet(u))));
console.log(results.map((r) => r.statusCode).join(" "), `in ${((Date.now() - t0) / 1000).toFixed(1)} s`);

Output:

200 200 200 200 200 200 in 6.5 s

Use one limiter per domain, not one for the whole job. For more on 429 handling, see how to handle rate limiting when scraping.

Step 6: route through a residential proxy

Datacenter IP ranges score worse than residential ones, as the language-neutral guide explains. Node's built-in fetch takes a dispatcher, and undici's ProxyAgent sends the request through a proxy.

// Step 6: route requests through a proxy with undici's ProxyAgent.
// PROXY_URL looks like http://user:pass@residential.example.com:8000
import { fetch, ProxyAgent } from "undici";

const dispatcher = new ProxyAgent(process.env.PROXY_URL);
const res = await fetch("https://httpbin.org/ip", { dispatcher, signal: AbortSignal.timeout(30_000) });
console.log(res.status, await res.text());

Output, through a local test proxy (the IP is hidden here):

200 {
  "origin": "[the proxy's exit IP]"
}

Import fetch from the undici package when you pass a dispatcher, so the two come from one version. With got-scraping, pass proxyUrl instead. A proxy changes your IP, not your TLS handshake, so a residential IP behind plain fetch still sends Node's fingerprint from Step 2.

Step 7: render JavaScript pages with Playwright

Some pages send an empty shell and build the content in the browser. For those, use a real browser. Automated Chromium sets navigator.webdriver to true. The stealth plugin patches that and other leaks, and playwright-extra lets Playwright use it.

// Step 7: render JavaScript pages with Playwright and the stealth plugin.
import { chromium } from "playwright-extra";
import stealth from "puppeteer-extra-plugin-stealth";

chromium.use(stealth());

const browser = await chromium.launch({ headless: true });
const context = await browser.newContext({ locale: "en-US", viewport: { width: 1366, height: 768 } });
const page = await context.newPage();

await page.goto("https://quotes.toscrape.com/js/", { waitUntil: "domcontentloaded" });
await page.waitForSelector(".quote", { timeout: 15_000 });
const quotes = await page.$$eval(".quote .text", (els) => els.map((e) => e.textContent));
console.log("navigator.webdriver:", await page.evaluate(() => navigator.webdriver));
console.log(quotes.length, "quotes, first:", quotes[0].slice(0, 60));

await browser.close();

Output:

navigator.webdriver: false
10 quotes, first: “The world as we have created it is a process of our thinkin

A browser costs real memory. In spring 2026 a developer on r/vibecoding described a Node and Puppeteer-Stealth setup on a 1 GB VPS where Chromium was eating the RAM, with 403s on a Cloudflare site and proof-of-work challenges on an Akamai site, even with residential proxies (Reddit). Use the browser only for pages that need it, and block images and fonts with page.route().

Step 8: send what still fails to an API

After Steps 1 to 7, some pages still fail. In our test, 46 of 100 pages came back on none of the four Node clients. This step keeps your direct path and sends only the blocked pages to String's fetch endpoint. looksBlocked() is the function from Step 1.

// Step 8: send the pages that still fail to String's fetch endpoint.
import { gotScraping } from "got-scraping";

async function stringFetch(url) {
  const res = await fetch("https://request.usestring.ai/v1/fetch", {
    method: "POST",
    headers: { Authorization: `Bearer ${process.env.STRING_API_KEY}`, "Content-Type": "application/json" },
    body: JSON.stringify({ url, format: "markdown", markdownMode: "readable" }),
    signal: AbortSignal.timeout(90_000),
  });
  // A 4xx or 5xx here is String's own error (bad key, rate limit, unreachable site), not the page's.
  if (!res.ok) throw new Error(`String API ${res.status}: ${await res.text()}`);
  return { status: Number(res.headers.get("x-status-code")), text: await res.text() };
}

export async function scrape(url, mustContain) {
  const direct = await gotScraping({ url, timeout: { request: 30_000 }, throwHttpErrors: false });
  const wall = looksBlocked(direct.statusCode, direct.body, mustContain);
  if (!wall) return { via: "direct", status: direct.statusCode, text: direct.body };
  const api = await stringFetch(url);
  return { via: `String (direct: ${wall})`, ...api };
}

for (const [url, mustContain] of [
  ["https://books.toscrape.com/", "A Light in the Attic"],
  ["https://www.homedepot.com/p/Milwaukee-M18-FUEL-18-Volt-Lithium-Ion-Brushless-Cordless-Gen-II-18-Gauge-Brad-Nailer-Tool-Only-2746-20/309752194", "Brad Nailer"],
]) {
  const r = await scrape(url, mustContain);
  console.log(new URL(url).hostname, r.status, r.via, r.text.includes(mustContain) ? "real page" : "not the page", `${r.text.length} chars`);
}

Output:

books.toscrape.com 200 direct real page 51274 chars
www.homedepot.com 200 String (direct: HTTP 403) real page 25944 chars

The site's own status comes back in the x-status-code header, and the page comes back as Markdown (response formats). To try it on one of your own blocked URLs from a shell:

curl https://request.usestring.ai/v1/fetch \
  -H "Authorization: Bearer $STRING_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"url": "https://www.homedepot.com/p/309752194", "format": "markdown"}'

How to avoid getting blocked in Node.js

The checklist, in the order that paid off in our test:

• Check the body for page text on every response. A 200 is not proof (Step 1).
• Do not put a Chrome User-Agent on plain fetch or axios and stop there. In our test it made things worse (Step 2).
• Use a client that sends Chrome-consistent headers over HTTP/2, such as got-scraping (Step 3).
• Keep one cookie jar and one IP per session (Steps 4 and 6).
• Cap concurrency per domain and honor Retry-After (Step 5).
• Use residential IPs for protected sites. Our test ran from a home IP, and a cloud server's IP will usually do worse (Step 6).
• Render in Playwright only the pages that build content in the browser (Step 7).
• Raise Node's header limit with --max-http-header-size if you see UND_ERR_HEADERS_OVERFLOW. Two large sites in our test sent over 16 KB of headers.
• Send what still fails to an API, and keep the direct path for the rest (Step 8).

For the reasons behind each technique, and the point where each one breaks, see the language-neutral guide. For vendor-specific walls, see how to bypass Cloudflare and how to get past Akamai Bot Manager.

How String handles blocked Node.js requests

String is the most reliable web scraping API, bypassing anti-bot, and you only pay when content comes back.

• One fetch call replaces the proxy pool and the browser. POST https://request.usestring.ai/v1/fetch picks a plain request or a browser, and a proxy tier, per site (fetch docs).
• String returned 86 of 100 protected pages by the marker rule, 90 after a hand check, in one call each on October 20, 2026. The best plain Node client in the same test, got-scraping, returned 50.
• String returned 39 of the 46 pages that no Node client got on October 20, 2026. In the September 16, 2026 benchmark run, String returned 40 of those 46 on all five attempts.
• String returned all 15 Cloudflare sites in the October 20, 2026 test. Plain Node fetch returned none of them.
• String had the highest success rate of 16 scraping APIs in the September 16, 2026 benchmark run: 97.0% across 100 sites. The field ran from 36.4% to 97.0% (benchmark).
• The response header x-billed-request-type names the fetch type each call used. In our test, 47 calls ran as standard requests, 42 as premium-proxy requests and 3 in a browser (fetch docs).

What it costs

Our 100-page test made 92 String calls that came back with a page. At Growth rates they cost about $0.10 in total: 47 standard requests at $0.20 per 1,000, 42 premium-proxy requests at $2.00, 2 browser requests at $1.00 and 1 browser request with premium proxies at $4.00. At Starter rates the same 92 calls cost about $0.15. Both fit inside the 5,000 free standard requests a new account gets (pricing).

The direct path in Steps 1 to 7 costs nothing per request. A residential proxy plan and the servers for a Playwright fleet do cost money, and the engineering time to keep them working costs more. Step 8 lets you pay only for the pages the direct path loses.

Start with the URLs that fail in your logs today. Run Step 8 on them with a free key, and you will know your block rate before you change your scraper.

FAQ

How do I scrape a website in Node.js without getting blocked?

Use a client whose TLS handshake and headers match one browser, such as got-scraping, then add one cookie jar per session, a concurrency cap with backoff, residential IPs, and Playwright for JavaScript pages. Check every body for page text. On October 20, 2026, plain Node fetch got 40 of 100 protected pages, got-scraping got 50, and String's fetch API got 86 by the same rule (90 after a hand check of four pages).

Why do I get a 403 in Node.js when the same request works in Python or the browser?

The server reads the TLS handshake before any header. Node's fetch offered HTTP/1.1 and 52 cipher suites in our test, which is not what Chrome sends (Step 2). Python's requests has a different handshake, and some sites score the two differently. Copying browser headers does not change the handshake.

Does setting a Chrome User-Agent in Node fetch stop blocks?

Not on its own. In our October 20, 2026 test, a full Chrome header set on Node fetch got 38 of 100 protected pages against 40 with the default headers. The headers claimed Chrome while the handshake said Node.

Is got-scraping better than axios or node-fetch for scraping?

On protected sites it was in our test. got-scraping 4.2.2 got 50 of 100 pages and plain Node fetch got 40. We did not test axios or node-fetch. Both run on Node's built-in TLS module, so neither sends a Chrome handshake.

Does Puppeteer or Playwright with a stealth plugin still work in 2026?

It works for pages that need JavaScript and for basic automation checks such as navigator.webdriver, which it set to false in our run. It does not fix IP reputation, and it costs a browser per page. Developers report 403s and proof-of-work challenges on Cloudflare and Akamai sites even with stealth and residential proxies.

How do I use a residential proxy with Node.js fetch?

Create an undici ProxyAgent with your proxy URL and pass it as dispatcher to fetch, importing both from the undici package. Step 6 shows the code. The proxy changes your IP. It does not change Node's TLS fingerprint.

How do I tell if a 200 response is a block page?

Test the body for text that only the real page carries, and for challenge strings such as "Just a moment..." or "Robot or human?". In our test, 9 responses on the got-scraping rung were 2xx without the page. Walmart, for example, returned a 15 KB "Robot or human?" page with status 200 to every Node client.

What is the best Node.js library for web scraping?

For fetching protected pages, got-scraping was the best plain Node client in our test. For parsing HTML, cheerio. For pages that build content in the browser, Playwright, with playwright-extra and the stealth plugin. For the pages none of those get, a fetch API such as String.

Sources

• Node.js block test, October 20, 2026: one GET per page per client to the 100 Web Data Frontier Benchmark target pages, run with Node 24.16.0, got-scraping 4.2.2, impit 0.14.6, p-limit 7.3.3, tough-cookie 6.0.2, undici 8.12.3, playwright 1.64.1, playwright-extra 4.3.6 and puppeteer-extra-plugin-stealth 2.11.2.
• Web Data Frontier Benchmark, September 16, 2026 run: 16 APIs, 100 sites, five attempts per site. Harness and raw results.
• Cloudflare: JA3/JA4 fingerprint
• FoxIO: JA4 technical details
• tls.peet.ws fingerprint echo
• got-scraping on GitHub
• impit on GitHub
• undici ProxyAgent docs
• Node.js CLI: --max-http-header-size
• tough-cookie on npm
• p-limit on npm
• playwright-extra on npm
• puppeteer-extra-plugin-stealth on npm
• Playwright: network routing
• MDN: Navigator.webdriver
• Reddit r/webscraping: Scraping api gets 403 in Node.js, but works fine in Python
• undici discussion #2249: Blocked by cloudflare?
• Stack Overflow: Why can't I fetch a webpage with Node.js and node-fetch?
• Reddit r/vibecoding: Seeking advice on bypassing Akamai/Cloudflare
• String fetch overview, fetch API reference, response formats, rate limits, MCP server, AI SDK guide
• String pricing, read October 20, 2026