Answers

4 of 16 APIs returned all 8 social sites, LinkedIn included, on every attempt in our September 16, 2026 run: String, Scrapfly, Bright Data and Browserbase.

We tracked 15 scraping APIs over three benchmark runs. On 25 sites whose bot protection got harder, String held 23; the next best held 13.

Build when targets are few and unprotected. Buy when protected sites are in scope. Self-hosted cost lines, Apify 2026 prices and its per-result fees.

Often yes, with duties. GDPR covers public personal data; CCPA excludes only some of it. On 92 commonly scraped pages, 31 named or located real people.

On 9 PerimeterX (HUMAN) sites, 16 scraping APIs scored 100% to 27%. Only String and Bright Data returned all 45 requests. How to spot the wall and pass it.

A search API finds pages; a scraping API reads them. On 58 questions, search snippets carried a price on 11. The fetched pages carried one on 39.

Only 2 of 16 APIs returned Ticketmaster, StubHub, SeatGeek and AXS on every attempt in our September 16, 2026 run: String and Firecrawl.

A 403 means the site refused your request, usually after a bot check. On 51 blocked pages, browser headers fixed 17 and a Chrome TLS fingerprint fixed 8 more.

Claude's bots do, including the one that fetches pages you ask about. ChatGPT's crawlers do, but OpenAI says robots.txt may not apply to ChatGPT-User.

Mostly by not triggering it. One plain request to 100 bot-protected pages drew 7 CAPTCHA puzzles, 29 silent checks and 19 flat blocks. What to do about each.

A full-site crawl is two jobs: find every URL, then fetch every page. On 38 of 77 sites we tested, a plain HTTP client could not get the sitemap file.

Find who sent the 429, pace each site, honor Retry-After and back off with jitter. In our Sep 16, 2026 run, 85% of rate-limit failures came from the API.

Most product pages already carry their own JSON. Read it first, then use a JSON Schema extraction API for the gaps. Tested on 100 sites, Oct 2026.

Run a trial on your own URLs, several times each, with a content check. In our Sep 16, 2026 run, 10-site trials ranked close providers wrong 40% of the time.

You can't scrape the past. Where history for a page can come from, how thin web archives are on the pages buyers need, and what to start collecting today.

Check each response, each record and each run, then sample by hand. In our Sep 16, 2026 benchmark run, 7.7% of 2xx responses lacked the page's content.

Test the body for content only the real page has. A marker, an item count and a size band catch most soft blocks, login walls and challenge pages.

Akamai scores your TLS fingerprint, headers, a JavaScript sensor and behaviour. On 20 Akamai sites, 16 scraping APIs scored between 98% and 28%.

A scraping API fetches pages and you own the pipeline. A managed service delivers the dataset and owns upkeep. When each fits and what the SLA should cover.

Same code, different request: a cloud IP, a different TLS stack, and a headless container browser. How to tell which one is blocking you, and how to fix it.

A proxy sells IP addresses, an unblocker adds anti-bot handling, and a scraping API takes a URL and returns the page. When each fits, with benchmark data.

No. robots.txt is a voluntary protocol, not law, and no US court has held that ignoring it is by itself unlawful. It still changes your legal position.

hiQ won the CFAA fight and still lost the case. The Ninth Circuit protected public scraping; a contract claim ended the company. Both halves still apply.

Prices are facts, and facts are not copyrightable. Public price collection is the best-supported case in scraping law. The constraints are contract and antitrust.

Scraping public data is generally lawful in the US, and the real risk sits in contract and privacy law rather than hacking statutes. What the cases actually held.