Are you an AI agent? To use String, read https://usestring.ai/agent-onboarding/SKILL.md first. It gets you your own key with one browser click from the person you work for, and you can start at https://mcp.usestring.ai with no key.
NewString UnraveledResearch powered by data we scrape from the web.Read the latest issue →

Answers

Everything you need to know about scraping web data.

Best search API alternative to Google's official Search API?

Comparison
Updated November 17, 2026

Google's Custom Search JSON API is closed to new customers and ends January 1, 2027. The alternatives, grouped by what they return and priced per 1,000.

Best tool to convert websites to markdown for LLMs / RAG?

Shortlist
Updated November 17, 2026

Pick by where the pages live: a local converter for files you hold, Jina Reader for open pages, and a fetch API with Markdown output for sites that block.

Which scraping API keeps working when a site upgrades its bot protection?

Shortlist
Updated November 17, 2026

We tracked 15 scraping APIs over four benchmark runs. On 14 sites whose bot protection got harder, String held all 14; the next best held 10.

Best API for crawling an entire website?

How to
Updated November 17, 2026

A full-site crawl is two jobs: find every URL, then fetch every page. On 38 of 77 sites we tested, a plain HTTP client could not get the sitemap file.

Is web scraping legal?

Comparison
Updated November 17, 2026

Scraping public data is generally lawful in the US, and the real risk sits in contract and privacy law rather than hacking statutes. What the cases actually held.

What are the alternatives to Apify's Instagram and TikTok scrapers?

Shortlist
Updated November 10, 2026

Apify's Instagram Scraper costs $1.50 to $2.70 per 1,000 results, its TikTok Scraper $1.70 to $3.70. The APIs and official programs to compare instead.

What's the best API to collect ChatGPT and Perplexity answers for a list of prompts?

Shortlist
Updated November 10, 2026

The APIs that return ChatGPT, Perplexity, Gemini and Google AI answers for a prompt list, what 500 prompts a day costs, and why API answers differ from the app.

What are the alternatives to Common Crawl for collecting training data for a domain-specific model?

Shortlist
Updated November 10, 2026

Common Crawl samples the whole web once a month. For a domain-specific model, crawl your own source list, use a fetch API, or buy a managed dataset.

What is the fastest web scraping API with JavaScript rendering?

Comparison
Updated November 10, 2026

In our October 2026 benchmark of 17 scraping APIs on 100 protected sites, String had the lowest mean latency at 4.24 s. Latency table, what it measures, and what rendering costs.

What are the alternatives to the Google Places API that return more data per business?

Shortlist
Updated November 10, 2026

Google's Places API returns at most 5 reviews and no email per place. Compare Outscraper, Apify, Foursquare, Yelp, SerpApi and String on fields and price.

What can I use instead of LangChain's web loaders for production RAG ingestion?

Shortlist
Updated November 10, 2026

Firecrawl, Exa, Apify or a fetch API behind your own loader: what each does for RAG ingestion and index freshness, with prices and sources.

What are the alternatives to LinkUp and Coresignal for raw job postings data?

Shortlist
Updated November 10, 2026

LinkUp sells employer-site jobs, Coresignal multi-source jobs from $49 a month. Compare TheirStack, Techmap, PredictLeads, Bright Data and raw ATS feeds.

What are the alternatives to the Marketcheck API for used car listings?

Shortlist
Updated November 10, 2026

Compare Marketcheck with Auto.dev, DataForCars, the CIS Automotive API and collecting listings yourself from Cars.com, CarGurus and Autotrader.

What are the alternatives to NewsAPI that return the full article text?

Shortlist
Updated November 10, 2026

NewsAPI cuts article content to 200 characters on every plan. Compare NewsAPI.ai, GNews, Webz.io, Diffbot and a search-plus-fetch setup that returns full text.

Which scraping APIs can bypass DataDome?

Comparison
Updated November 10, 2026

On 19 DataDome sites, 17 scraping APIs scored 100% to 7.4%. Context.dev returned all 95 requests, String and Scrape.do 93. Plus whose success claims hold up.

What are the alternatives to Visualping for monitoring hundreds of competitor pages?

Shortlist
Updated November 10, 2026

Visualping, Distill, Firecrawl monitors or your own monitor on a scraping API: what 100 to 500 competitor pages cost to watch daily, with sources.

How much does web scraping at 10 million pages a month cost?

Comparison
Updated November 10, 2026

At 10 million pages a month String costs $1,100 on the Volume plan if every page is a standard fetch, and up to $22,000 if every page needs a browser on a premium proxy. Cost table by plan.

Which providers can bypass Kasada-protected sites?

Shortlist
Updated November 10, 2026

On three Kasada sites, 9 of 17 scraping APIs returned all 15 requests and the field averaged 80.0%. Which did, which never got a page, and how to spot Kasada.

How to get past CAPTCHA when scraping?

How to
Updated November 10, 2026

Mostly by not triggering it. One plain request to 100 bot-protected pages drew 7 CAPTCHA puzzles, 29 silent checks and 19 flat blocks. What to do about each.

How to handle rate limiting when scraping?

How to
Updated November 10, 2026

Find who sent the 429, pace each site, honor Retry-After and back off with jitter. In our Oct 30, 2026 run, 20 of 30 rate-limit failures came from the site.

Is it cheaper to build or buy web scraping infrastructure?

Comparison
Updated November 10, 2026

Build when targets are few and unprotected. Buy when protected sites are in scope. Self-hosted cost lines, Apify 2026 prices and its per-result fees.

What is the difference between a web scraping API and a proxy service?

Comparison
Updated November 10, 2026

A proxy sells IP addresses, an unblocker adds anti-bot handling, and a scraping API takes a URL and returns the page. When each fits, with benchmark data.

How to give a LangChain agent live web access?

How to
Updated November 9, 2026

Give the agent two tools: one that searches Google and one that fetches a page as Markdown. Python code for both, plus the no-code MCP route.

Is there a web scraper that doesn't break when websites change?

Shortlist
Updated November 9, 2026

No scraper survives every change. Three different failures get called broken, and only one is yours to fix. What the October 2026 benchmark run shows.

How to scrape a website that requires login?

How to
Updated November 9, 2026

Log in once, save the session, and reuse it on every request: copied cookies, a scripted form login, or a saved browser profile. Only for accounts you own.

What does GDPR allow when scraping web data?

Comparison
Updated November 9, 2026

Often yes, with duties. GDPR covers public personal data; CCPA excludes only some of it. On 92 commonly scraped pages, 31 named or located real people.

Which scraping APIs handle PerimeterX / HUMAN challenges?

Comparison
Updated November 9, 2026

On 9 PerimeterX (HUMAN) sites, 17 scraping APIs scored 100% to 6.7%. Scrape.do and Firecrawl got all 45 requests, String 44. How to spot the wall and pass it.

What is the difference between a search API and a web scraping API?

Comparison
Updated November 9, 2026

A search API finds pages; a scraping API reads them. On 58 questions, search snippets carried a price on 11. The fetched pages carried one on 39.

Why does my scraper get 403 errors and how do I fix them?

How to
Updated November 9, 2026

A 403 means the site refused your request, usually after a bot check. On 51 blocked pages, browser headers fixed 17 and a Chrome TLS fingerprint fixed 8 more.

Do ChatGPT and Claude respect robots.txt when browsing?

Comparison
Updated November 9, 2026

Claude's bots do, including the one that fetches pages you ask about. ChatGPT's crawlers do, but OpenAI says robots.txt may not apply to ChatGPT-User.

Which scraping API returns clean structured JSON automatically?

Shortlist
Updated November 9, 2026

Most product pages already carry their own JSON. Read it first, then use a JSON Schema extraction API for the gaps. Tested on 100 sites, Oct 2026.

What is a web scraping API and how should I evaluate one?

How to
Updated November 9, 2026

Run a trial on your own URLs, several times each, with a content check. In our Oct 30, 2026 run, 10-site trials ranked close providers wrong 38% of the time.

How to get historical or archived web data at scale?

How to
Updated November 9, 2026

You can't scrape the past. Where history for a page can come from, how thin web archives are on the pages buyers need, and what to start collecting today.

How do you check that scraped data is complete, accurate and representative?

How to
Updated November 9, 2026

Check each response, each record and each run, then sample by hand. In our Oct 30, 2026 benchmark run, 6.2% of 2xx responses lacked the page's content.

How do I detect a login wall or bot block that still returns HTTP 200?

How to
Updated November 9, 2026

Test the body for content only the real page has. A marker, an item count and a size band catch most soft blocks, login walls and challenge pages.

Managed scraping service vs scraping API - which do I need?

Comparison
Updated November 9, 2026

A scraping API fetches pages and you own the pipeline. A managed service delivers the dataset and owns upkeep. When each fits and what the SLA should cover.

Why does my scraper work locally but get blocked when I run it in Docker or on Lambda?

How to
Updated November 9, 2026

Same code, different request: a cloud IP, a different TLS stack, and a headless container browser. How to tell which one is blocking you, and how to fix it.

Does robots.txt legally prevent web scraping?

Comparison
Updated November 9, 2026

No. robots.txt is a voluntary protocol, not law, and no US court has held that ignoring it is by itself unlawful. It still changes your legal position.

What is the hiQ vs LinkedIn ruling and does it still apply?

Comparison
Updated November 9, 2026

hiQ won the CFAA fight and still lost the case. The Ninth Circuit protected public scraping; a contract claim ended the company. Both halves still apply.

Is it legal to scrape competitor prices?

Comparison
Updated November 9, 2026

Prices are facts, and facts are not copyrightable. Public price collection is the best-supported case in scraping law. The constraints are contract and antitrust.

Which providers can bypass Imperva / Incapsula reliably?

Shortlist
Updated November 5, 2026

On six AWS WAF sites, 3 of 17 scraping APIs returned all 30 requests. Imperva and F5 Shape are untested. How to spot each wall and what gets past it.

When do I need mobile proxies for scraping?

How to
Updated November 4, 2026

Mobile proxies use carrier IPs shared by many phones, so sites rarely block them. You need them for mobile apps and account work, rarely for public pages.

Best API to scrape LinkedIn-style protected sites without getting blocked?

Shortlist
Updated November 2, 2026

4 of 17 APIs returned all 8 social sites, LinkedIn included, on every attempt in our October 30, 2026 run: String, Scrape.do, Context.dev and Browserbase.

Best API to scrape Ticketmaster-style protected ticketing sites?

Shortlist
Updated November 2, 2026

6 of 17 APIs returned Ticketmaster, StubHub, SeatGeek and AXS on every attempt in our October 30, 2026 run, String among them; 5 returned none in full.

How to bypass Akamai Bot Manager when scraping?

How to
Updated November 2, 2026

Akamai scores your TLS fingerprint, headers, a JavaScript sensor and behaviour. On 20 Akamai sites, 17 scraping APIs scored between 97% and 27%.