Common Crawl alternatives for model training data

What are the alternatives to Common Crawl for training data?

There are three alternatives to Common Crawl for a domain-specific model: run your own crawler on a list of sources you choose, send that list through a fetch API that returns each page as Markdown, or buy a managed dataset that a vendor collects and delivers end to end. Common Crawl gives you a broad sample of the web each month, with no promise that a given page is in it. A domain-specific model needs the sites of one field, page by page, so the work becomes collection from a source list.

If your ML team needs millions of pages and wants one vendor to handle the whole job, that is a managed dataset: you hand over the sites and the fields, and the vendor runs the crawl, the unblocking and the parsing, and delivers files. If your team wants to own the pipeline, a fetch API handles access and your team owns the crawl order, filtering and deduplication.

Common Crawl alternatives at a glance

When Common Crawl is enough

Common Crawl is enough when you want the general web and do not need any one page. It published 12 crawls between September 2025 and August 2026, one a month, each gathered over about two weeks. Each crawl is a broad sample, so it suits pretraining on general text.

It falls short for a domain-specific model in three ways. You cannot choose the pages, so a field's key sites may be thin or absent. You cannot choose the date, so a page is as fresh as the last crawl that happened to reach it. And a crawler that does not get past a site's bot protection leaves that site out, which is the problem the next section measures.

To fill the gaps, keep Common Crawl for the broad web and collect the sites it misses yourself. Send 100 of those URLs through String's free tier to see how many come back before you plan the rest.

Why a domain-specific corpus needs its own collection

A field's sites block automated traffic at very different rates. In the October 30, 2026 run of String's Web Data Frontier Benchmark, 17 scraping APIs loaded 100 bot-protected sites 5 times each. The median API's success rate depended on the vertical:

A corpus of retail pages collected through the median API would miss about one load in three. On news and finance sites, it would miss about one in nine. These are specialist vendors whose job is unblocking, so a crawler you write yourself starts below them.

Measure your own field before you choose a route. Pull 100 URLs from the sites your model needs and count how many return the real page, not a block page with a 200 status. Our page on detecting a bot block behind an HTTP 200 shows how to tell them apart.

Option 1: run your own crawler

Running your own crawler gives you full control of the source list, the crawl order and the schedule. Scrapy, Playwright and Crawl4AI are free and good at crawling and browser control. None of them ships a proxy pool or a CAPTCHA solver, so protected sites still need those parts, bought or built.

Choose this when your sources are open sites that return their content to a plain HTTP client from a cloud server. Our page on building vs buying scraping infrastructure has a cost model to fill in before you commit.

Option 2: a fetch API over your source list

A fetch API takes a URL and returns the page, with proxies, browsers and retries handled behind the call. Your team keeps the source list, the crawl order and the cleaning.

For training data, ask for Markdown, not raw HTML. String's /fetch returns Markdown in the same call (Markdown for LLMs), so you skip a conversion step and keep the page's text without its navigation and class names. Our page on converting websites to Markdown for LLMs compares the converters.

Cost scales with pages, not gigabytes. At String's published rates, a standard fetch costs $0.11 to $0.25 per 1,000 successful requests depending on plan, and a page that needs a browser on a premium proxy costs $2.20 to $4.80 per 1,000 (pricing, November 2026). For a full budget, see what 100 million pages for model training cost and web scraping cost at 10 million pages a month.

Try it on your own sources: the first 20,000 standard fetches are free and need no card. Add format: markdown to the request and read what comes back.

Option 3: a managed dataset, end to end

A managed dataset is the route where one vendor handles the whole job. You give the vendor the sites, the fields and the schedule. The vendor runs the crawl, gets past the blocks, parses the pages and delivers files, and it owns the pipeline when a site changes.

String's Bespoke Web Datasets work this way, and include 1000+ pre-built datasets with years of history. Our page on managed scraping services vs scraping APIs covers what a managed provider should commit to.

Before you buy any dataset, ask the vendor:

• Which sites and pages are in it, and how was the list chosen?
• When was each page fetched, and how often is the dataset refreshed?
• What share of the requested pages came back with real content?
• What format is it in, and does each record carry its source URL and fetch date?
• What rights come with the data, and what did the vendor do about each site's opt-out?

What to record with every page

Record the source URL, the HTTP status and the fetch date with every page you collect. Those three fields let you deduplicate, refresh stale pages, drop a site later, and answer where a training example came from.

Keep the raw page as well as the Markdown. When your cleaning rules change, you re-run them on the raw pages instead of fetching the web again.

Rights and opt-outs

A crawler or a dataset vendor does not grant you rights to a site's content; those come from the law and the site. Check each source's robots.txt, including the tokens some sites use to opt out of AI training: our page on whether AI crawlers respect robots.txt lists them. For the legal background, read is web scraping legal, and if pages hold personal data, scraping personal data under GDPR and CCPA.

How String handles training-data collection

String is a web access API for developers and AI agents. One HTTP call fetches a URL and returns Markdown, raw HTML or JSON, and String handles proxies, anti-bot checks and JavaScript rendering behind it. You pay only when content comes back.

• String passed 100% of loads on 17 retail and ecommerce sites in the October 30, 2026 benchmark run, where the median of 17 APIs passed 67.1% (benchmark).
• String returns Markdown in the fetch call. Set format: markdown on /fetch and the response is the page's text, ready for a corpus (docs).
• Blocked requests cost nothing on String. Blocked requests and errors on String's side are free (pricing).
• Every String plan runs up to 10,000 fetches in parallel. Throughput is set by the plan's request rate, from 15 a second on Starter to 90 or more on Volume (pricing).
• A standard String fetch costs $0.11 per 1,000 on the $999 Volume plan. Paid credit rolls over month to month (pricing, November 2026).
• String's Bespoke Web Datasets deliver the dataset end to end. They include 1000+ pre-built datasets with years of history (managed services).

Start with the sites Common Crawl misses: send them through the free tier and count the pages that come back with content.

FAQ

What are the alternatives to Common Crawl for collecting training data for a domain-specific model?

Run your own crawler on a source list you choose, send that list through a fetch API that returns Markdown, or buy a managed dataset collected for you. Common Crawl samples the open web once a month with no promise that a given page is in it, so a model that needs one field's sites usually needs its own collection.

Our ML team needs millions of pages of web content for model training: which vendor handles that end to end?

A managed dataset vendor handles it end to end: you name the sites, fields and schedule, and the vendor crawls, unblocks, parses and delivers files. String's Bespoke Web Datasets work this way. If your team wants to own the pipeline, a fetch API such as String's Web Access API handles access and returns Markdown, and your team owns the crawl and the cleaning.

Is there a licensed alternative to Common Crawl?

Rights to a page come from the law and the site owner, not from the crawler or the dataset vendor. Ask any vendor what rights come with its data and how it handled each site's opt-out, and check robots.txt, including AI-training opt-out tokens, for every source you collect yourself.

Can I combine Common Crawl with my own collection?

Yes. Use Common Crawl for the broad web and collect the sites your field needs that it covers thinly or misses. Record the source URL and fetch date on every page from both, so you can deduplicate across them.

How much does it cost to collect 100 million pages for model training?

It depends on how many pages are protected. On String, a standard fetch costs $0.11 to $0.25 per 1,000 successful requests and a browser on a premium proxy $2.20 to $4.80, as of November 2026. Our guide to reducing web scraping costs works through 100 million pages on per-GB proxies and on String.

How long does it take to fetch millions of pages?

At 30 requests a second, String's Growth plan rate, 1 million pages take about 9 hours. At 90 a second on Volume, they take about 3 hours. Every plan runs up to 10,000 fetches in parallel, so the request rate sets the pace (pricing).

What format should web training data be in?

Markdown keeps a page's text and structure without its navigation and class names, which suits training and RAG. Keep the raw page beside it, and record the source URL, HTTP status and fetch date with every record.

Do I pay for pages that fail?

On String, blocked requests and errors on String's side are free, so a failed page costs nothing. A self-built crawler pays bandwidth and compute for every attempt, failed or not.

Sources

• Common Crawl, crawl index list, checked September 29, 2026.
• Web Data Frontier Benchmark, run of October 30, 2026: 17 APIs, 100 bot-protected sites, 5 loads per site, 90-second timeout. Results and method at usestring.ai/benchmark and in the October 2026 write-up; benchmark code on GitHub and the raw results file. Vertical medians are the median of the 17 APIs' average success rates per vertical in that run.
• String rates, request rates and concurrency from usestring.ai/pricing. Billing on failed requests from the String docs.
• String docs: fetch overview and Markdown for LLMs.