--- title: "How to scrape news articles" canonical: "https://usestring.ai/use-cases/news-scraping" description: "The String Web Access API is the best way to scrape news articles. It returns any article page, or just its headline, authors, publish time and section as JSON, from news sites that block scrapers, like Reuters, Bloomberg and The Wall Street Journal." updated: 2026-10-19 --- > Markdown version of https://usestring.ai/use-cases/news-scraping, for AI agents. Reading this yourself? The full page is at https://usestring.ai/use-cases/news-scraping. # How to scrape news articles The String Web Access API is the best way to scrape news articles. It returns any article page, or just its headline, authors, publish time and section as JSON, from news sites that block scrapers, like Reuters, Bloomberg and The Wall Street Journal. In our [September 2026 benchmark](https://usestring.ai/benchmark) it fetched seven news sites on every attempt, one of two providers out of 16 to do so, and returned 97% of requests across 100 bot-protected sites, the most of any provider. **On this page** 1. [Quickstart](#quickstart) 2. [Why scrape news sites?](#why-scrape-news-sites) 3. [Why does news scraping break?](#why-does-news-scraping-break) 4. [How do I build a news feed from sites that block scrapers?](#how-do-i-build-a-news-feed-from-sites-that-block-scrapers) 5. [Get a free sample news feed](#get-a-free-sample-news-feed) 6. [Frequently asked questions](#frequently-asked-questions) --- ## Quickstart ### From your agent Add the hosted MCP server to Claude, Cursor, ChatGPT or any MCP client. Then give your agent an article URL. It calls the `web_access_fetch` tool and reads the article as Markdown. ```text https://mcp.usestring.ai/v1/mcp ``` [MCP setup](https://portal.usestring.ai/docs/mcp/remote) · [n8n](https://portal.usestring.ai/docs/integrations/n8n) · [Zapier](https://portal.usestring.ai/docs/integrations/zapier) ### From the API One call returns a Reuters article's headline, authors, times and section as JSON. If the page has no headline or publish time, you get `"extracted": false` back instead of a guess. ```bash curl https://request.usestring.ai/v1/fetch \ -H "Authorization: Bearer $STRING_API_KEY" \ -H "Content-Type: application/json" \ -d '{ "url": "https://www.reuters.com/business/energy/supertanker-rates-fall-record-highs-2026-10-16/", "countryCode": "US", "jsonSchema": { "type": "object", "properties": { "headline": { "type": "string" }, "authors": { "type": "array", "items": { "type": "string" } }, "published": { "type": "string" }, "modified": { "type": ["string", "null"] }, "section": { "type": ["string", "null"] }, "publisher": { "type": "string" } }, "required": ["headline", "published", "publisher"] } }' ``` Response (run with an API key on October 19, 2026; `x-json-schema-applied: true`, billed `request_premium`): ```json { "headline": "Supertanker rates fall from record highs", "authors": ["Enes Tunagur"], "published": "2026-10-16T14:19:40.03Z", "modified": "2026-10-16T14:25:41.382Z", "section": "Energy", "publisher": "Reuters" } ``` [Get a free API key](https://portal.usestring.ai/sign-up) · [Structured extraction docs](https://portal.usestring.ai/docs/fetch/structured-extraction) --- ## Why scrape news sites? News sites publish around the clock, and the articles that matter to you are spread across many publishers. A news feed fetches new articles from the sites you follow, reads each one, and keeps a clean record you can search, alert on or hand to a model. It answers the questions teams ask most: - **What is being written about us?** Coverage of your company, your customers and your competitors, as it's published, so the people who need to respond read it first. - **What just happened that moves our numbers?** Deals, sanctions, outages and earnings reach Reuters and Bloomberg before they reach a dataset. Research and trading teams need them with the publish time attached. - **Does our model know about it yet?** An LLM or search index that reads yesterday's news answers questions about today with stale facts. ### What you can collect | What | Examples | |---|---| | Article records | Headline, authors, publish and update times, section and publisher, from each article's schema.org article data | | Article text | The text the page serves to a logged-out visitor, as clean Markdown for search, summaries or retrieval | | New articles | Fresh URLs from section pages, publisher sitemaps and date-limited search results, checked as often as you need | | Updates and paywall status | Changed headlines and update times on articles you already have, and the publisher's own flag for whether an article is free to read | --- ## Why does news scraping break? - **News sites block scrapers.** In our [September 2026 benchmark](https://usestring.ai/benchmark), 9 of the 15 other scraping APIs couldn't fetch reuters.com on every attempt, and 8 of 15 couldn't fetch wsj.com. Apify, Bright Data, ScrapingBee, Browserbase and ScrapingAnt got 0 of 5 on reuters.com. Across Bloomberg, Reuters, WSJ, The New York Times, Barron's, MarketWatch and Yahoo Finance, only String and ScraperAPI returned all seven on every attempt. See all 16 providers in [the best web scraping APIs](https://usestring.ai/comparisons/best-web-scraping-apis). - **Headlines change after publishing.** The Wall Street Journal listed one story as "Antofagasta Shares Slump on Weak Copper Outlook" in search results, while the article's own data carried "Antofagasta Raises Cost Outlook Due to Global Diesel Prices". A feed keyed on headlines stores the same story twice. - **A blocked page looks like an article.** The page loads with a challenge or an empty shell, and your feed saves an article with no headline or date. Benchmark rows for the seven news sites (successful attempts out of 5, September 16, 2026 run): | Site | String | Scrapfly | ScraperAPI | Firecrawl | Apify | Bright Data | ScrapingBee | Context.dev | Oxylabs | Nimble | Zyte | Decodo | Scrapingdog | Browserbase | ZenRows | ScrapingAnt | |---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---| | bloomberg.com | 5 | 4 | 5 | 5 | 5 | 5 | 5 | 4 | 5 | 5 | 5 | 5 | 1 | 4 | 0 | 4 | | reuters.com | 5 | 3 | 5 | 5 | 0 | 0 | 0 | 5 | 4 | 5 | 5 | 2 | 3 | 0 | 5 | 0 | | wsj.com | 5 | 5 | 5 | 5 | 5 | 4 | 0 | 0 | 5 | 4 | 0 | 2 | 5 | 0 | 5 | 0 | | nytimes.com | 5 | 5 | 5 | 0 | 5 | 5 | 5 | 5 | 5 | 5 | 5 | 5 | 2 | 5 | 5 | 0 | | barrons.com | 5 | 5 | 5 | 5 | 5 | 5 | 5 | 5 | 4 | 5 | 5 | 5 | 1 | 0 | 5 | 0 | | marketwatch.com | 5 | 5 | 5 | 5 | 5 | 5 | 5 | 5 | 5 | 5 | 5 | 4 | 0 | 5 | 5 | 0 | | finance.yahoo.com | 5 | 4 | 5 | 5 | 5 | 0 | 0 | 5 | 5 | 5 | 5 | 5 | 0 | 5 | 0 | 0 | --- ## Why do teams choose String? - **It fetches the most pages.** String returned 97% of requests across 100 bot-protected sites in our [September 2026 benchmark](https://usestring.ai/benchmark), the most of 16 providers, and all 35 attempts on the seven news sites. - **You pay only for pages that come back.** Blocks and errors are free. - **There's nothing to maintain.** You don't run proxies, fingerprints or captcha solvers. - **Empty pages don't become articles.** Mark the headline and publish time as required, and a page that comes back without them is flagged instead of saved. --- ## How do I build a news feed from sites that block scrapers? ### 1. Find new article URLs Poll the section pages and article sitemaps of the publishers you follow, and keep every URL you haven't seen. Section pages list the newest articles first, so you only need to read the top of each one. **With String:** The `/search` endpoint returns live Google results for a query like "site:reuters.com/business", limited to the past day with `dateRange` and sorted newest first. A sitemap crawl lists every URL on a publisher's domain. ### 2. Fetch each article once, by its canonical URL Strip tracking parameters such as `?mod=` before you store a URL. The same article reaches you from search, section pages and newsletters with different query strings. **With String:** Blocked requests are retried, and you don't pay for the ones that fail. Set `countryCode` when a publisher serves a different edition by country. ### 3. Read the record from the article's own data Reuters, WSJ, MarketWatch and Yahoo Finance put a schema.org NewsArticle in each article page, and Bloomberg an Article, with the headline, authors, publish and update times. It's more reliable than the visible page, and it survives redesigns that break CSS selectors. ### 4. Treat a missing headline or date as a failure An article with no headline or publish time was probably a blocked or changed page. Flag it and fetch it again later, so the gap never lands in your feed. ### 5. Dedupe on the URL, not the headline Publishers rewrite headlines after an article goes out. Key each record on the canonical URL and keep the latest headline as a field. ### 6. Strip invisible characters before you compare text A Reuters article we fetched on October 19 carried 51 zero-width spaces and 11 word joiners inside its text. Remove them before you hash, search or compare article text, or two copies of the same paragraph won't match. ### 7. Re-check recent articles for updates Publishers update articles after they go out. The Reuters story in the quickstart was updated six minutes after it was published. Fetch articles from the last day again and keep the new version when the update time changes. --- ## Get a free sample news feed Tell us which publishers and topics you follow. We'll run String on their pages and email you a sample feed with headline, authors, publish time and section, free. Email [support@usestring.ai](mailto:support@usestring.ai) with your work email, what you want to follow and the news sites or section URLs, or use the [form on the page](https://usestring.ai/use-cases/news-scraping#sample-feed). What a news feed looks like (rows read from each article's schema.org data in String fetches on October 19, 2026): | Publisher | Headline | Authors | Published (UTC) | |---|---|---|---| | Reuters | French markets tumble as riots add to political crisis | Alun John | 2026-10-19 15:12 | | Reuters | Supertanker rates fall from record highs | Enes Tunagur | 2026-10-16 14:19 | | Bloomberg | UK Watchdog Proposes Tougher Scrutiny of Microsoft Software | Upmanyu Trivedi | 2026-10-16 09:04 | | Yahoo Finance | CoTec, BMR plan partnership to boost Punitaqui copper production | Shree Mishra | 2026-10-16 11:02 | | WSJ | Antofagasta Raises Cost Outlook Due to Global Diesel Prices | Aimee Look | 2026-10-15 06:50 | --- ## Frequently asked questions ### What's the best way to scrape news articles? Find new article URLs from section pages, sitemaps or date-limited search, fetch each article once by its canonical URL, and read the record from the page's schema.org article data. The hard part is getting the pages, because major news sites block most scrapers. The String Web Access API returned 97% of requests across 100 bot-protected sites in our [September 2026 benchmark](https://usestring.ai/benchmark), the most of 16 providers, and fetched Bloomberg, Reuters, WSJ, The New York Times, Barron's, MarketWatch and Yahoo Finance on every attempt. ### How do I scrape Reuters articles? Fetch the article URL with String and read the schema.org NewsArticle data in the page: headline, authors, publish and update times, and section. Reuters was the hardest news site in our September 2026 benchmark: 9 of the 15 other scraping APIs couldn't fetch it on every attempt, and five got nothing back. String fetched it on all 5 attempts. Reuters article pages billed at String's premium tier on our fetches. ### How much does it cost to scrape news articles? The Reuters articles we fetched billed at String's premium tier, which is $3.00 per 1,000 successful requests on the Starter plan ($20 a month) and $2.00 per 1,000 on Growth ($100 a month). A feed of 500 articles a day is about 15,000 requests a month, or $45 in usage on Starter at that rate. Structured extraction with `jsonSchema` adds a per-request fee that scales with page size. The first 5,000 standard requests are free, and failed requests aren't billed. See [pricing](https://usestring.ai/pricing). ### Is it legal to scrape news articles? Collecting facts from public news pages, like headlines, authors and publish times, is the easier case. Article text is copyrighted, so storing it for internal search or analysis is a different question from republishing it, and many publishers' terms restrict both. Stay on public pages, keep your request rate reasonable, don't republish article text, and check with your counsel. We go through the cases in [Is web scraping legal?](https://usestring.ai/answers/is-web-scraping-legal) ### Can String scrape paywalled articles? String fetches the page a logged-out visitor gets and doesn't log in, so you get what the publisher serves without a subscription. Reuters, Bloomberg, WSJ and MarketWatch mark their articles with `isAccessibleForFree: false` in the page data, so your feed can record which articles are behind a paywall. ### How do I get only new articles? Keep a list of the canonical URLs you've already fetched, and fetch only URLs that aren't on it. Find candidates from section pages and article sitemaps, or run String's `/search` endpoint with `dateRange` set to the past day and `sortBy` set to date. ### Can I feed news articles into an LLM or RAG pipeline? String returns any page as clean Markdown when you set `format` to `markdown`, and the MCP tool returns Markdown by default. It's ready to chunk and embed. Store the headline, publish time and URL from the article's data with each chunk, so answers can cite the source and prefer the newest article. ### How do I avoid getting blocked when scraping news sites? Send each request to String instead of the site. Proxy rotation, browser fingerprints, captcha solving and retries happen behind the endpoint, and JavaScript rendering is switched on per request for pages that need it. Requests that still fail aren't billed. ### Does String work on the news sites I need? Send us the URLs and we'll run them and show you what comes back before you commit. Ask for a [free sample news feed](#get-a-free-sample-news-feed), or test them yourself with your first 5,000 free requests. --- ## Send us the sites you can't get We'll show you what comes back. [Get started](https://portal.usestring.ai/sign-up) · [Book a demo](https://usestring.ai/book-demo) All use cases: https://usestring.ai/use-cases --- [llms.txt](https://usestring.ai/llms.txt): machine-readable index of this site for AI agents.