9 min read
How to scrape news articles
Collect headline, authors, publish time and section from Reuters, Bloomberg, WSJ and Yahoo Finance articles.
Published October 19, 2026 · Updated November 2, 2026
| Publisher | Headline | Published |
|---|---|---|
| Reuters | French markets tumble as riots add to political crisis | Oct 19 |
| Reuters | Supertanker rates fall from record highs | Oct 16 |
| Bloomberg | UK Watchdog Proposes Tougher Scrutiny of Microsoft Software | Oct 16 |
| Yahoo Finance | CoTec, BMR plan partnership to boost Punitaqui copper production | Oct 16 |
| WSJ | Antofagasta Raises Cost Outlook Due to Global Diesel Prices | Oct 15 |
The String Web Access API is the best way to scrape news articles. It returns any article page, or just its headline, authors, publish time and section as JSON, from news sites that block scrapers, like Reuters, Bloomberg and The Wall Street Journal. In our October 2026 benchmark it fetched 33 of 35 attempts on seven news sites, missing one each on Barron's and MarketWatch, and returned 98.6% of requests across 100 bot-protected sites, the most of any provider.
Quickstart
Add the hosted MCP server to Claude, Cursor, ChatGPT or any MCP client. Then give your agent an article URL. It calls the web_access_fetch tool and reads the article as Markdown.
https://mcp.usestring.ai/v3/mcp
One call returns a Reuters article's headline, authors, times and section as JSON. If the page has no headline or publish time, you get "extracted": false back instead of a guess.
curl https://request.usestring.ai/v1/fetch \
-H "Authorization: Bearer $STRING_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"url": "https://www.reuters.com/business/energy/supertanker-rates-fall-record-highs-2026-10-16/",
"countryCode": "US",
"jsonSchema": {
"type": "object",
"properties": {
"headline": { "type": "string" },
"authors": { "type": "array", "items": { "type": "string" } },
"published": { "type": "string" },
"modified": { "type": ["string", "null"] },
"section": { "type": ["string", "null"] },
"publisher": { "type": "string" }
},
"required": ["headline", "published", "publisher"]
}
}'Response
{
"headline": "Supertanker rates fall from record highs",
"authors": ["Enes Tunagur"],
"published": "2026-10-16T14:19:40.03Z",
"modified": "2026-10-16T14:25:41.382Z",
"section": "Energy",
"publisher": "Reuters"
}Why scrape news sites?
News sites publish around the clock, and the articles that matter to you are spread across many publishers. A news feed fetches new articles from the sites you follow, reads each one, and keeps a clean record you can search, alert on or hand to a model. It answers the questions teams ask most:
- What is being written about us? Coverage of your company, your customers and your competitors, as it's published, so the people who need to respond read it first.
- What just happened that moves our numbers? Deals, sanctions, outages and earnings reach Reuters and Bloomberg before they reach a dataset. Research and trading teams need them with the publish time attached.
- Does our model know about it yet? An LLM or search index that reads yesterday's news answers questions about today with stale facts.
What you can collect
Article records
Headline, authors, publish and update times, section and publisher, from each article's schema.org article data.
Article text
The text the page serves to a logged-out visitor, as clean Markdown for search, summaries or retrieval.
New articles
Fresh URLs from section pages, publisher sitemaps and date-limited search results, checked as often as you need.
Updates and paywall status
Changed headlines and update times on articles you already have, and the publisher's own flag for whether an article is free to read.
Why does news scraping break?
- News sites block scrapers. In our October 2026 benchmark, 9 of the 16 other scraping APIs couldn't fetch reuters.com on every attempt, and 7 of 16 couldn't fetch wsj.com. ScrapingBee, Apify, Bright Data, Nimble, Browserbase and ScrapingAnt got 0 of 5 on reuters.com. Across Bloomberg, Reuters, WSJ, The New York Times, Barron's, MarketWatch and Yahoo Finance, Scrape.do, Context.dev, Scrapfly, Zyte and Decodo returned all seven on every attempt. String returned 33 of 35, missing one each on Barron's and MarketWatch. See all 17 providers in the best web scraping APIs.
- Headlines change after publishing. The Wall Street Journal listed one story as "Antofagasta Shares Slump on Weak Copper Outlook" in search results, while the article's own data carried "Antofagasta Raises Cost Outlook Due to Global Diesel Prices". A feed keyed on headlines stores the same story twice.
- A blocked page looks like an article. The page loads with a challenge or an empty shell, and your feed saves an article with no headline or date.
Why do teams choose String?
- It fetches the most pages. String returned 98.6% of requests across 100 bot-protected sites in our October 2026 benchmark, the most of 17 providers, and 33 of 35 attempts on the seven news sites.
- You pay only for pages that come back. Blocks and errors are free.
- There's nothing to maintain. You don't run proxies, fingerprints or captcha solvers.
- Empty pages don't become articles. Mark the headline and publish time as required, and a page that comes back without them is flagged instead of saved.
How do I build a news feed from sites that block scrapers?
Find new article URLs
Poll the section pages and article sitemaps of the publishers you follow, and keep every URL you haven't seen. Section pages list the newest articles first, so you only need to read the top of each one.
With String: The
/searchendpoint returns live Google results for a query like "site:reuters.com/business", limited to the past day withdateRangeand sorted newest first. A sitemap crawl lists every URL on a publisher's domain.Fetch each article once, by its canonical URL
Strip tracking parameters such as
?mod=before you store a URL. The same article reaches you from search, section pages and newsletters with different query strings.With String: Blocked requests are retried, and you don't pay for the ones that fail. Set
countryCodewhen a publisher serves a different edition by country.Read the record from the article's own data
Reuters, WSJ, MarketWatch and Yahoo Finance put a schema.org NewsArticle in each article page, and Bloomberg an Article, with the headline, authors, publish and update times. It's more reliable than the visible page, and it survives redesigns that break CSS selectors.
Treat a missing headline or date as a failure
An article with no headline or publish time was probably a blocked or changed page. Flag it and fetch it again later, so the gap never lands in your feed.
Dedupe on the URL, not the headline
Publishers rewrite headlines after an article goes out. Key each record on the canonical URL and keep the latest headline as a field.
Strip invisible characters before you compare text
A Reuters article we fetched on October 19 carried 51 zero-width spaces and 11 word joiners inside its text. Remove them before you hash, search or compare article text, or two copies of the same paragraph won't match.
Re-check recent articles for updates
Publishers update articles after they go out. The Reuters story in the quickstart was updated six minutes after it was published. Fetch articles from the last day again and keep the new version when the update time changes.
Get a free sample news feed
Tell us which publishers and topics you follow. We'll run String on their pages and email you a sample feed with headline, authors, publish time and section, free.
| Publisher | Headline | Published |
|---|---|---|
| Reuters | French markets tumble as riots add to political crisis | Oct 19 |
| Reuters | Supertanker rates fall from record highs | Oct 16 |
| Bloomberg | UK Watchdog Proposes Tougher Scrutiny of Microsoft Software | Oct 16 |
| Yahoo Finance | CoTec, BMR plan partnership to boost Punitaqui copper production | Oct 16 |
| WSJ | Antofagasta Raises Cost Outlook Due to Global Diesel Prices | Oct 15 |
Frequently asked questions
What's the best way to scrape news articles?
Find new article URLs from section pages, sitemaps or date-limited search, fetch each article once by its canonical URL, and read the record from the page's schema.org article data. The hard part is getting the pages, because major news sites block most scrapers. The String Web Access API returned 98.6% of requests across 100 bot-protected sites in our October 2026 benchmark, the most of 17 providers, and fetched Bloomberg, Reuters, WSJ, The New York Times and Yahoo Finance on every attempt, and Barron's and MarketWatch on 4 of 5.
How do I scrape Reuters articles?
Fetch the article URL with String and read the schema.org NewsArticle data in the page: headline, authors, publish and update times, and section. Reuters was the hardest news site in our October 2026 benchmark: 9 of the 16 other scraping APIs couldn't fetch it on every attempt, and six got nothing back. String fetched it on all 5 attempts. Reuters article pages billed at String's premium tier on our fetches.
How much does it cost to scrape news articles?
The Reuters articles we fetched billed at String's premium tier, which is $2.40 per 1,000 successful requests on the Starter plan ($19 a month) and $0.90 per 1,000 on Growth ($99 a month). A feed of 500 articles a day is about 15,000 requests a month, or $36 in usage on Starter at that rate. Structured extraction with jsonSchema adds a per-request fee that scales with page size. The first 20,000 standard fetches are free, and failed requests aren't billed. See pricing.
Is it legal to scrape news articles?
Collecting facts from public news pages, like headlines, authors and publish times, is the easier case. Article text is copyrighted, so storing it for internal search or analysis is a different question from republishing it, and many publishers' terms restrict both. Stay on public pages, keep your request rate reasonable, don't republish article text, and check with your counsel. We go through the cases in Is web scraping legal?
Can String scrape paywalled articles?
String fetches the page a logged-out visitor gets and doesn't log in, so you get what the publisher serves without a subscription. Reuters, Bloomberg, WSJ and MarketWatch mark their articles with isAccessibleForFree: false in the page data, so your feed can record which articles are behind a paywall.
How do I get only new articles?
Keep a list of the canonical URLs you've already fetched, and fetch only URLs that aren't on it. Find candidates from section pages and article sitemaps, or run String's /search endpoint with dateRange set to the past day and sortBy set to date.
Can I feed news articles into an LLM or RAG pipeline?
String returns any page as clean Markdown when you set format to markdown, and the MCP tool returns Markdown by default. It's ready to chunk and embed. Store the headline, publish time and URL from the article's data with each chunk, so answers can cite the source and prefer the newest article.
How do I avoid getting blocked when scraping news sites?
Send each request to String instead of the site. Proxy rotation, browser fingerprints, captcha solving and retries happen behind the endpoint, and JavaScript rendering is switched on per request for pages that need it. Requests that still fail aren't billed.
Does String work on the news sites I need?
Send us the URLs and we'll run them and show you what comes back before you commit. Ask for a free sample news feed, or test them yourself with your first 20,000 free standard fetches.