Answers
Everything you need to know about scraping web data.
Best search API alternative to Google's official Search API?
Updated November 17, 2026
Google's Custom Search JSON API is closed to new customers and ends January 1, 2027. The alternatives, grouped by what they return and priced per 1,000.
Best tool to convert websites to markdown for LLMs / RAG?
Updated November 17, 2026
Pick by where the pages live: a local converter for files you hold, Jina Reader for open pages, and a fetch API with Markdown output for sites that block.
Which scraping API keeps working when a site upgrades its bot protection?
Updated November 17, 2026
We tracked 15 scraping APIs over four benchmark runs. On 14 sites whose bot protection got harder, String held all 14; the next best held 10.
Best API for crawling an entire website?
Updated November 17, 2026
A full-site crawl is two jobs: find every URL, then fetch every page. On 38 of 77 sites we tested, a plain HTTP client could not get the sitemap file.
Is web scraping legal?
Updated November 17, 2026
Scraping public data is generally lawful in the US, and the real risk sits in contract and privacy law rather than hacking statutes. What the cases actually held.
What are the alternatives to Apify's Instagram and TikTok scrapers?
Updated November 10, 2026
Apify's Instagram Scraper costs $1.50 to $2.70 per 1,000 results, its TikTok Scraper $1.70 to $3.70. The APIs and official programs to compare instead.
What's the best API to collect ChatGPT and Perplexity answers for a list of prompts?
Updated November 10, 2026
The APIs that return ChatGPT, Perplexity, Gemini and Google AI answers for a prompt list, what 500 prompts a day costs, and why API answers differ from the app.
What are the alternatives to Common Crawl for collecting training data for a domain-specific model?
Updated November 10, 2026
Common Crawl samples the whole web once a month. For a domain-specific model, crawl your own source list, use a fetch API, or buy a managed dataset.
What is the fastest web scraping API with JavaScript rendering?
Updated November 10, 2026
In our October 2026 benchmark of 17 scraping APIs on 100 protected sites, String had the lowest mean latency at 4.24 s. Latency table, what it measures, and what rendering costs.
What are the alternatives to the Google Places API that return more data per business?
Updated November 10, 2026
Google's Places API returns at most 5 reviews and no email per place. Compare Outscraper, Apify, Foursquare, Yelp, SerpApi and String on fields and price.
What can I use instead of LangChain's web loaders for production RAG ingestion?
Updated November 10, 2026
Firecrawl, Exa, Apify or a fetch API behind your own loader: what each does for RAG ingestion and index freshness, with prices and sources.
What are the alternatives to LinkUp and Coresignal for raw job postings data?
Updated November 10, 2026
LinkUp sells employer-site jobs, Coresignal multi-source jobs from $49 a month. Compare TheirStack, Techmap, PredictLeads, Bright Data and raw ATS feeds.
What are the alternatives to the Marketcheck API for used car listings?
Updated November 10, 2026
Compare Marketcheck with Auto.dev, DataForCars, the CIS Automotive API and collecting listings yourself from Cars.com, CarGurus and Autotrader.
What are the alternatives to NewsAPI that return the full article text?
Updated November 10, 2026
NewsAPI cuts article content to 200 characters on every plan. Compare NewsAPI.ai, GNews, Webz.io, Diffbot and a search-plus-fetch setup that returns full text.
Which scraping APIs can bypass DataDome?
Updated November 10, 2026
On 19 DataDome sites, 17 scraping APIs scored 100% to 7.4%. Context.dev returned all 95 requests, String and Scrape.do 93. Plus whose success claims hold up.
What are the alternatives to Visualping for monitoring hundreds of competitor pages?
Updated November 10, 2026
Visualping, Distill, Firecrawl monitors or your own monitor on a scraping API: what 100 to 500 competitor pages cost to watch daily, with sources.
How much does web scraping at 10 million pages a month cost?
Updated November 10, 2026
At 10 million pages a month String costs $1,100 on the Volume plan if every page is a standard fetch, and up to $22,000 if every page needs a browser on a premium proxy. Cost table by plan.
Which providers can bypass Kasada-protected sites?
Updated November 10, 2026
On three Kasada sites, 9 of 17 scraping APIs returned all 15 requests and the field averaged 80.0%. Which did, which never got a page, and how to spot Kasada.
How to get past CAPTCHA when scraping?
Updated November 10, 2026
Mostly by not triggering it. One plain request to 100 bot-protected pages drew 7 CAPTCHA puzzles, 29 silent checks and 19 flat blocks. What to do about each.
How to handle rate limiting when scraping?
Updated November 10, 2026
Find who sent the 429, pace each site, honor Retry-After and back off with jitter. In our Oct 30, 2026 run, 20 of 30 rate-limit failures came from the site.
Is it cheaper to build or buy web scraping infrastructure?
Updated November 10, 2026
Build when targets are few and unprotected. Buy when protected sites are in scope. Self-hosted cost lines, Apify 2026 prices and its per-result fees.
What is the difference between a web scraping API and a proxy service?
Updated November 10, 2026
A proxy sells IP addresses, an unblocker adds anti-bot handling, and a scraping API takes a URL and returns the page. When each fits, with benchmark data.
How to give a LangChain agent live web access?
Updated November 9, 2026
Give the agent two tools: one that searches Google and one that fetches a page as Markdown. Python code for both, plus the no-code MCP route.
Is there a web scraper that doesn't break when websites change?
Updated November 9, 2026
No scraper survives every change. Three different failures get called broken, and only one is yours to fix. What the October 2026 benchmark run shows.
How to scrape a website that requires login?
Updated November 9, 2026
Log in once, save the session, and reuse it on every request: copied cookies, a scripted form login, or a saved browser profile. Only for accounts you own.
What does GDPR allow when scraping web data?
Updated November 9, 2026
Often yes, with duties. GDPR covers public personal data; CCPA excludes only some of it. On 92 commonly scraped pages, 31 named or located real people.
Which scraping APIs handle PerimeterX / HUMAN challenges?
Updated November 9, 2026
On 9 PerimeterX (HUMAN) sites, 17 scraping APIs scored 100% to 6.7%. Scrape.do and Firecrawl got all 45 requests, String 44. How to spot the wall and pass it.
What is the difference between a search API and a web scraping API?
Updated November 9, 2026
A search API finds pages; a scraping API reads them. On 58 questions, search snippets carried a price on 11. The fetched pages carried one on 39.
Why does my scraper get 403 errors and how do I fix them?
Updated November 9, 2026
A 403 means the site refused your request, usually after a bot check. On 51 blocked pages, browser headers fixed 17 and a Chrome TLS fingerprint fixed 8 more.
Do ChatGPT and Claude respect robots.txt when browsing?
Updated November 9, 2026
Claude's bots do, including the one that fetches pages you ask about. ChatGPT's crawlers do, but OpenAI says robots.txt may not apply to ChatGPT-User.
Which scraping API returns clean structured JSON automatically?
Updated November 9, 2026
Most product pages already carry their own JSON. Read it first, then use a JSON Schema extraction API for the gaps. Tested on 100 sites, Oct 2026.
What is a web scraping API and how should I evaluate one?
Updated November 9, 2026
Run a trial on your own URLs, several times each, with a content check. In our Oct 30, 2026 run, 10-site trials ranked close providers wrong 38% of the time.
How to get historical or archived web data at scale?
Updated November 9, 2026
You can't scrape the past. Where history for a page can come from, how thin web archives are on the pages buyers need, and what to start collecting today.
How do you check that scraped data is complete, accurate and representative?
Updated November 9, 2026
Check each response, each record and each run, then sample by hand. In our Oct 30, 2026 benchmark run, 6.2% of 2xx responses lacked the page's content.
How do I detect a login wall or bot block that still returns HTTP 200?
Updated November 9, 2026
Test the body for content only the real page has. A marker, an item count and a size band catch most soft blocks, login walls and challenge pages.
Managed scraping service vs scraping API - which do I need?
Updated November 9, 2026
A scraping API fetches pages and you own the pipeline. A managed service delivers the dataset and owns upkeep. When each fits and what the SLA should cover.
Why does my scraper work locally but get blocked when I run it in Docker or on Lambda?
Updated November 9, 2026
Same code, different request: a cloud IP, a different TLS stack, and a headless container browser. How to tell which one is blocking you, and how to fix it.
Does robots.txt legally prevent web scraping?
Updated November 9, 2026
No. robots.txt is a voluntary protocol, not law, and no US court has held that ignoring it is by itself unlawful. It still changes your legal position.
What is the hiQ vs LinkedIn ruling and does it still apply?
Updated November 9, 2026
hiQ won the CFAA fight and still lost the case. The Ninth Circuit protected public scraping; a contract claim ended the company. Both halves still apply.
Is it legal to scrape competitor prices?
Updated November 9, 2026
Prices are facts, and facts are not copyrightable. Public price collection is the best-supported case in scraping law. The constraints are contract and antitrust.
Which providers can bypass Imperva / Incapsula reliably?
Updated November 5, 2026
On six AWS WAF sites, 3 of 17 scraping APIs returned all 30 requests. Imperva and F5 Shape are untested. How to spot each wall and what gets past it.
When do I need mobile proxies for scraping?
Updated November 4, 2026
Mobile proxies use carrier IPs shared by many phones, so sites rarely block them. You need them for mobile apps and account work, rarely for public pages.
Best API to scrape LinkedIn-style protected sites without getting blocked?
Updated November 2, 2026
4 of 17 APIs returned all 8 social sites, LinkedIn included, on every attempt in our October 30, 2026 run: String, Scrape.do, Context.dev and Browserbase.
Best API to scrape Ticketmaster-style protected ticketing sites?
Updated November 2, 2026
6 of 17 APIs returned Ticketmaster, StubHub, SeatGeek and AXS on every attempt in our October 30, 2026 run, String among them; 5 returned none in full.
How to bypass Akamai Bot Manager when scraping?
Updated November 2, 2026
Akamai scores your TLS fingerprint, headers, a JavaScript sensor and behaviour. On 20 Akamai sites, 17 scraping APIs scored between 97% and 27%.