Build vs buy web scraping infrastructure: 2026 costs

Should you build or buy web scraping infrastructure?

Build web scraping infrastructure when your targets are few, unprotected and stable, and scraping is part of what you sell. Buy when bot-protected sites are in scope, because that is where an in-house stack spends most of its money.

The fetch code is cheap. The cost sits in proxies, browsers, solvers, and the engineer who repairs the pipeline every time a target changes its defences. In String's Web Data Frontier Benchmark (October 30, 2026 run), 11 of the 100 bot-protected sites defeated most of the 17 scraping APIs tested, and those vendors do unblocking full time.

The decision rule

Count your targets and check what protects them. If every target returns its content to a plain HTTP client from a cloud server, build: an open-source framework and a small server will do, and no vendor adds much. If even a few targets sit behind Cloudflare, DataDome, Akamai, PerimeterX or Kasada, price the build honestly with the model below before you commit, because the protected sites set the cost of the whole system.

Scrapy, Playwright and Crawl4AI are free and good. None of them ships a proxy pool or a CAPTCHA solver. You buy or build those parts separately, and you own the result when they stop working.

Can an LLM write and maintain your scrapers?

An LLM can write and repair your scraper code. It does not get the request past a site that blocks it, so on bot-protected sites you still need proxies, browsers and unblocking, built or bought.

A scraper has two halves. The first is parsing: selectors, pagination and field mapping. A coding model such as Claude or ChatGPT writes that half quickly and rewrites it when a layout changes, if you hand it the new HTML. The second half is access: IP addresses the site trusts, a browser that passes fingerprint checks, and challenge pages handled. Access is infrastructure you run and pay for, and it is the half that fails on protected sites.

The benchmark shows where the failures sit. In the October 30, 2026 run, 17 scraping APIs loaded 100 bot-protected sites 5 times each, and on 11 sites more than half of the APIs failed most of their loads. Those vendors run code, proxies and browsers full time. A scraper a model wrote from scratch starts with no proxies at all.

Upkeep is where a model helps least on its own. It fixes a broken selector once someone notices the break and gives it the new page. Spotting the break, re-running the pages you missed and keeping a pool of clean IPs stay with your team. To stop layout changes breaking the parser at all, ask for Markdown or a JSON schema instead of class names, as our page on building a scraper that doesn't break shows.

The split most teams land on:

• Let the model write and fix parsers, pagination and field mapping.
• Buy access for protected sites: a scraping API returns the page, and the model's code parses it.
• Build access yourself only when every target returns its content to a plain HTTP client from a cloud server.
• Give an agent a fetch tool instead of a hand-written scraper. String's MCP server hands the agent one tool that returns the page.

Self-hosted web scraping infrastructure costs: a model to fill in

Put your own numbers in each row and compare the monthly total against a vendor quote for the same volume and the same success rate.

Two rows decide most outcomes. Maintenance is a standing cost, and it grows with the number of protected targets. See what happens when a site upgrades its protection across three benchmark runs. Failure cost is invisible until you measure it: a site that passes three loads in five costs you five attempts for three pages, and each attempt burns bandwidth and compute.

For the buy column, use published rates. As of October 2026, String charges per 1,000 successful requests: from $0.11 to $4.80 depending on plan, fetch path (plain HTTP or a rendered browser) and proxy class, on plans that start at $19 a month, with 20,000 standard fetches free (pricing). Most failed requests are not billed.

To fill in the buy column with real numbers, run a sample of your own URLs on the free tier: no card, and you see the success rate before you pay.

Apify vs building your own scraper: which is cheaper long term?

Apify's site-specific scrapers charged a median $2.70 per 1,000 results on the Starter plan when we checked Apify Store on October 28, 2026. Put that number against what a self-built scraper costs you each month. Building is cheaper long term only when servers, proxies and upkeep for your targets cost less than that bill at your volume.

Apify pricing and cost in 2026

Apify sells four plans. Prices are from apify.com/pricing, checked October 27, 2026, billed monthly:

Each plan's fee is prepaid usage. On paid plans, usage above it goes on the next invoice; the Free plan pauses until the next cycle. Unused usage expires at the end of each billing cycle. Most Store Actors add their own fee per result or per event on top.

What a site-specific Actor charges per result

We looked up the 92 sites in our benchmark outside social media in Apify Store on October 28, 2026. For each site we took the Actor with the site's name in its title and the most users in the last 30 days. 70 sites had a pay-per-result Actor.

• On October 28, 2026, the median price was $2.70 per 1,000 results at Starter rates. The middle half ran from $1.00 to $4.99.
• On October 28, 2026, 45 of the 70 charged more per result than Apify's own Web Fetch charges per page, $1.50 per 1,000 on Starter.
• On October 28, 2026, Indeed's most-used Actor charged $0.09 per 1,000 results, Zillow's $3.00, Booking's $3.90 and Amazon's $4.00.
• On October 28, 2026, 2 of the 70 carried Apify's "under maintenance" label, which Apify gives an Actor that fails its automated tests.

A result is whatever the Actor's author defines: a product, a review, a listing or a row. Compare it with a cost per page only when one page gives one result.

Where the long-term cost sits

Take 100,000 product pages a month, one result per page, on each route's entry plan, at String's rates as of October 2026 and Apify's as checked October 27, 2026:

To find your break-even, divide the monthly bill for the route you would buy by your loaded cost per engineer hour. The answer is how many hours of upkeep a month a self-built scraper can use before it costs more, with servers and proxies still unpaid.

Failed attempts change the math too. Apify's Web Fetch and String bill only successful requests, and a self-built stack pays for every attempt in bandwidth and compute. In the October 30, 2026 benchmark run, Apify's Web Fetch passed 79.4% of 500 attempts and String passed 98.6%. On 11 of the 92 sites outside social media, Web Fetch returned none of its 5 attempts, so a site-specific Actor is the Apify route there. String returned all 5 on 10 of those 11.

Builders do save money at first. One developer on r/scrapingtheweb moved Google Maps scraping in-house because "the API costs were getting ridiculous for my use case." A team that tried the same at scale replied: "In the end the saving was just not worth it especially when we added in proxy and dev time cost."

Test the buy side on your own targets before you decide: send 100 of your URLs through String's free tier and count the pages that come back with data.

Why the hard tail decides the answer

The benchmark sent 5 loads to each of 100 bot-protected sites through 17 APIs on October 30, 2026. A load passed only when the response contained a marker from the real page, so a CAPTCHA page with a 200 status counted as a failure.

• In the October 30, 2026 run, on 11 of the 100 sites, more than half of the 17 APIs failed most of their loads.
• In the October 30, 2026 run, on 12 sites, the average success rate across all 17 APIs was under 50%.
• No site passed every load on all 17 APIs.
• Temu defeated every API except String, which passed all 5 loads. In the same run, String passed every load on 9 of the 11 hard sites, the most of any API tested (next: 7). It passed 4 of 5 on Skyscanner and 2 of 5 on Kroger.

The spread by anti-bot system is wide. In the October 30, 2026 run, on the 15 Cloudflare sites, results ran from 6.7% to 98.7%. On the 19 DataDome sites, from 7.4% to 100%. On the 20 Akamai sites, from 27% to 97%. Those are specialist vendors. An in-house stack starts below them and has to catch up with the same fingerprinting and challenge work.

Practitioners say the same. A comment in r/webscraping put it this way: small scrapers for unprotected sites "are mostly gone or worth a lot less", but once scale or bot mitigation grows, "LLMs can't one or multiple-shot anything." AI coding tools made the easy part of scraping cheap. The hard part still costs what it did.

When building wins

• Fewer than a handful of targets, and none is bot-protected.
• The targets rarely change layout, and a day of stale data costs little.
• Scraping is your product or your moat, so owning the stack is the point.
• Very high, steady volume on a fixed set of easy sites, where a per-request rate adds up.
• A requirement forces you to own the network path end to end.

When buying wins

• Any meaningful share of targets sits behind a commercial anti-bot system.
• The target list grows or changes each quarter.
• Nobody on the team wants to be on call for a broken proxy pool.
• The data feeds a product, a model or an agent, and collection is plumbing.
• You need a success rate you can measure and put in a contract.

A common setup is hybrid: buy access (proxies, rendering, unblocking) through an API such as the Web Access API, and keep parsing and business logic in-house. If you want neither, a managed service delivers the dataset and owns the pipeline. Our page on managed scraping services vs scraping APIs covers that split.

How String keeps the buy column predictable

String is a web fetch API for public pages. It picks a plain request, a premium proxy or a real browser for each URL and bills only when the page comes back.

• String bills per successful request, not per result, gigabyte or compute unit. A standard fetch costs $0.16 per 1,000 on the $99 Growth plan, as of October 2026 (pricing).
• Blocked requests cost nothing on String. Blocked requests and errors on String's side are free; the two billed exceptions are a request with custom headers that the site blocks and a request to a site with an invalid certificate (pricing).
• String moves a request to a premium proxy only when the site blocks the standard one. Sites that work on the standard proxy stay at the standard rate (pricing).
• String passed 98.6% of 500 attempts on 100 bot-protected sites in the October 30, 2026 benchmark run, the highest of 17 APIs tested (benchmark).
• Paid String credit rolls over month to month. The $5 free credit covers 20,000 standard fetches and never expires (pricing).
• Every String plan runs up to 10,000 fetches in parallel. Throughput is set by the plan's request rate, from 15 a second on Starter (pricing).

What to ask a vendor

• How do you measure success: on page content or on HTTP status?
• What is your success rate on my targets? Run a trial on your own URL list, not the vendor's demo sites.
• Do I pay for blocked, empty or timed-out requests?
• What does the SLA cover, and what happens when you miss it?
• Where do you send my requests, what do you log, and how long do you keep it?
• What happens to my data and my code if I leave?

Next step: list your targets, fill in the cost model above, and run the buy column on String with your own URLs. The first 20,000 standard fetches are free and need no card (start here).

FAQ

Is it cheaper to build or buy web scraping infrastructure?

Building is cheaper when your targets are few, stable and unprotected. Buying is cheaper once bot-protected sites are in scope, because proxies, browser compute, CAPTCHA handling and maintenance make up most of an in-house stack's cost. In String's October 30, 2026 benchmark, 11 of 100 protected sites defeated most of the 17 APIs tested.

How should an engineering team decide between building scraping infrastructure and buying an API?

List the targets and check which anti-bot system protects each one. Fill in a monthly cost model for engineering, proxy bandwidth, browser compute, maintenance and failed attempts, then compare it with a vendor's rate at the same success rate on your own URLs. Build if the targets are easy and scraping is your product; buy access if protected sites are in scope.

Apify vs building my own scraper: which is cheaper long term?

Run the same cost model for both, and add Apify's fee per result. Apify Starter is $19 a month, and the most-used Store Actor for 70 of our benchmark sites charged a median $2.70 per 1,000 results on October 28, 2026. Divide the Apify bill by your hourly engineering cost to see how much upkeep the build can afford. Our Apify review and alternatives prices Apify's compute units and splits its benchmark score by site difficulty.

How much does Apify cost in 2026?

Free is $0 with $5 of usage a month. Starter is $19, Scale $199 and Business $999, and each fee is prepaid usage. Compute units cost $0.20 on Free and Starter, $0.16 on Scale and $0.13 on Business. Most Store Actors add a fee per result: the most-used Actor for 70 of our benchmark sites charged a median $2.70 per 1,000 results on Starter, checked October 28, 2026.

What does self-hosted web scraping infrastructure cost?

Four lines: servers, proxy bandwidth, browser compute for pages that need JavaScript, and engineer hours for upkeep. Scrapy, Playwright and Crawl4AI cost nothing to download. Upkeep grows with the number of protected targets, and you pay bandwidth and compute for every failed attempt. Price each line for your own targets with the cost model on this page.

Does Apify charge for failed scraping attempts?

It depends on the Actor's pricing model. Web Fetch charges one event per successful request and nothing for failed ones, plus a small start event per run. Pay-per-event Actors charge for the events their author defines, such as a result. Pay-per-usage Actors bill the compute and proxy time each run uses, whether or not it returns data.

Are open-source scraping frameworks enough?

Scrapy, Playwright and Crawl4AI are free and cover fetching, crawling and browser control. None of them ships a proxy pool or a CAPTCHA solver, so protected sites still need those parts, bought or built. On unprotected sites, an open-source framework is often all you need.

Can an LLM write and maintain my scrapers, or do I still need a scraping service?

An LLM can write and fix the parsing code: selectors, pagination and field mapping. It does not supply IP addresses a site trusts, a browser that passes fingerprint checks or challenge handling, and those decide whether a protected site returns the page. In String's October 30, 2026 benchmark, on 11 of 100 protected sites more than half of the 17 APIs tested failed most of their loads. Use the model for parsers and buy access for protected targets.

Sources

• Web Data Frontier Benchmark, run of October 30, 2026: 17 APIs, 100 bot-protected sites, 5 loads per site, 90-second timeout. Results and method at usestring.ai/benchmark and in the October 2026 write-up; harness on GitHub and the raw results file.
• String rates from usestring.ai/pricing, checked October 28, 2026. Billing on failed requests from the String docs.
• r/webscraping comment by u/AdaObvlada, August 25, 2026, permalink.
• Apify plans, compute-unit and Web Fetch rates from apify.com/pricing and the Web Fetch pricing page, checked October 27, 2026. Billing models from the Apify Store docs.
• Store Actor prices from Apify's public Store listing, read October 28, 2026: for each of the 92 benchmark sites outside social media, the Actor with the site's name in its title and the most users in the last 30 days, priced at its Starter-tier rate for the main result event.
• r/scrapingtheweb thread by u/Puzzled_Librarian_65 with a reply by u/NZRedditUser, thread.