Alternatives to LangChain web loaders for production RAG

What can I use instead of LangChain's web loaders for production RAG ingestion?

Use a hosted web data API behind a small loader of your own. If you crawl whole sites and want change diffs, use Firecrawl. If you need to find new documents by meaning, use Exa's search and contents. If your sources block bots, use a fetch API that returns Markdown and gets through protection, such as String. Keep LangChain for splitting, embedding and retrieval. LangChain's own docs call langchain-community, where WebBaseLoader lives, unmaintained, and mark community loaders as unverified.

Sources: LangChain document loader docs and the web_base reference, checked November 10, 2026.

Why teams replace LangChain's web loaders

WebBaseLoader and RecursiveUrlLoader are fine for a prototype. They fetch with a plain HTTP client from your own server, so a page behind Cloudflare, DataDome or Akamai comes back as a challenge page or an error. A challenge page that returns HTTP 200 gets split, embedded and retrieved like any other document.

The maintenance signal is plain. As of November 2026, LangChain's document loader page says community loaders are "user-contributed and unverified" and calls the langchain-community package unmaintained, with a link to its sunset notice (LangChain docs, checked November 10, 2026). The same page lists hosted alternatives under "Webpages": Firecrawl, Apify, Diffbot, Docling, Unstructured, Hyperbrowser and others.

Production ingestion needs four things a basic loader does not give you:

1. Pages that actually come back, including from sites that block data center traffic.
2. Clean text, as Markdown, so chunks hold content and not navigation.
3. A way to know what changed, so you re-embed only changed pages instead of the whole corpus.
4. A record of what failed, so a blocked page is a retry, not a silent gap in the index.

The options, side by side

Prices are as of November 2026, checked November 10, 2026 on each vendor's pricing page or docs, linked in the table.

Firecrawl vs Exa for keeping a RAG index fresh: which is better?

Firecrawl is better when you know the URLs and need to know which ones changed. Exa is better when the hard part is finding new documents on a topic. They solve different halves of freshness.

Firecrawl's change tracking compares each scrape with the last one for the same URL and labels the page new, same, changed or removed, with an optional line diff. Snapshots do not expire, so a page checked monthly still compares correctly (Firecrawl change tracking docs, checked November 10, 2026). That label is what an incremental re-index needs: re-embed the changed pages and delete the removed ones. Basic tracking and git-diff mode add no credits; JSON mode costs 5 credits a page. One catch: by default Firecrawl may return a cached copy up to 2 days old, and cached results still cost 1 credit (Firecrawl docs, checked November 10, 2026). Set maxAge lower when you need today's version.

Exa's Contents API serves pages from its own cache and crawls live when a page is missing or older than maxAgeHours. Each URL in the response carries a status and a source of cached or crawled, so you can tell which copy you got (Exa contents docs, checked November 10, 2026). Exa has no per-URL change label. You compare the text yourself. Its strength is the other half: search over its index finds pages you did not know about, and its Monitors run searches on a schedule for $2 per 1,000 requests (Exa pricing, checked November 10, 2026).

So the pick depends on the corpus:

• A fixed set of docs sites or help centers: Firecrawl's crawl with change tracking, or your own fetch plus a content hash.
• An open topic, such as news on a market or new research: Exa search to find documents, then fetch them.
• Sources that block bots: neither label helps if the page never comes back. Test your own source list first.

What freshness costs at 10,000 pages a day

Re-fetching 10,000 pages every day is 300,000 page fetches in a 30-day month. Prices below are list prices as of November 2026, checked November 10, 2026 (String as of October 2026).

Most corpora do not need a daily full pass. Re-fetch fast-moving pages daily and slow ones weekly, and compare a content hash before you re-embed. Embedding usually costs more than fetching once the corpus is large.

Which sources a loader actually gets back

A loader is only as good as the share of pages it returns. On the three developer and research sites in the October 30, 2026 run of String's open Web Data Frontier Benchmark, GitHub, Stack Overflow and arXiv, 7 of the 17 APIs tested returned all 15 attempts. The field averaged 76.5%. Firecrawl returned 66.7% there and 81.2% across all 100 sites, running below its current anti-bot setting. String returned 15 of 15 there and 98.6% across all 100 sites. Exa was not tested.

These are the kinds of sites a RAG corpus for coding agents is built from. A third of attempts failing on one of them means a third of a refresh pass silently keeps old text. Run the open benchmark code on your own source list before you choose, and see how to convert websites to Markdown for LLMs for the conversion side.

A loader you own, in 30 lines

This loader fetches each URL as Markdown through String, skips pages whose content has not changed since the last run, and returns LangChain Document objects for the rest. It reads STRING_API_KEY from the environment.

import hashlib
import json
import os
import pathlib

import requests
from langchain_core.documents import Document

STATE = pathlib.Path("hashes.json")

def load_changed(urls: list[str]) -> list[Document]:
    seen = json.loads(STATE.read_text()) if STATE.exists() else {}
    docs = []
    for url in urls:
        resp = requests.post(
            "https://request.usestring.ai/v1/fetch",
            headers={"Authorization": f"Bearer {os.environ['STRING_API_KEY']}"},
            json={"url": url, "format": "markdown"},
            timeout=150,
        )
        if resp.status_code != 200 or resp.headers.get("x-status-code", "200") != "200":
            print(f"retry later: {url} {resp.status_code}")
            continue
        digest = hashlib.sha256(resp.text.encode()).hexdigest()
        if seen.get(url) == digest:
            continue  # unchanged: keep the existing embeddings
        seen[url] = digest
        docs.append(Document(page_content=resp.text, metadata={"source": url}))
    STATE.write_text(json.dumps(seen))
    return docs

We have not run this exact loader end to end. The request fields and the x-status-code header, which carries the status the site itself returned, come from String's fetch docs, checked November 10, 2026. Pass the result to your text splitter and vector store as usual. To list a site's URLs first, String's sitemap jobs map a whole domain. For an agent that searches and reads at query time instead, see how to give a LangChain agent live web access.

How String handles RAG ingestion

String is a web data API for developers and AI agents that fetches, searches and maps any public site without getting blocked, and returns each page as HTML, Markdown or JSON. For a RAG pipeline, that means:

• String returned 493 of 500 requests on 100 protected sites, 98.6%, in the October 30, 2026 run of the open benchmark, first of 17 APIs.
• Markdown in one call. format: "markdown" returns clean text ready for a splitter (response formats).
• The site's own status comes back in a header. x-status-code tells a real 404 apart from a fetch failure, so deleted pages leave the index.
• A blocked request is not billed. String bills when the destination's response comes back (pricing docs, checked November 10, 2026).
• Discovery included. Google search and site maps run on the same key, so finding and fetching sources is one integration (search docs).

Where String fits

• For: RAG teams whose sources include sites that block bots, and teams that want to own the loader, the schedule and the change logic.
• Not for: teams that want a hosted crawler with change labels built in. Firecrawl's change tracking does that.
• Price: as of October 2026, Starter is $19 a month and Growth is $99 a month, with the fee credited as usage. A standard fetch is $0.25 per 1,000 on Starter and $0.16 on Growth (pricing). The first 20,000 standard fetches are free.
• Integrations: a REST API, an MCP server, n8n and Zapier.
• Limits: 15 requests a second on Starter and 30 on Growth, up to 10,000 fetches in flight (pricing, as of October 2026). No built-in change diff.

For more crawler options, see Firecrawl alternatives and how to crawl an entire website with an API.

FAQ

What can I use instead of LangChain's web loaders for production RAG ingestion?

A hosted web data API behind a small loader you own: Firecrawl for crawling with change diffs, Exa for finding new documents by search, or a fetch API such as String for sources that block bots. Keep LangChain for splitting and retrieval. As of November 2026, LangChain's docs call the langchain-community package unmaintained (LangChain docs, checked November 10, 2026).

Firecrawl vs Exa for keeping a RAG index fresh: which is better?

Firecrawl, when you know the URLs and need to know which changed: its change tracking labels each page new, same, changed or removed. Exa, when you need to find new documents: its search runs over its own index, and its Contents API can force a live fetch with maxAgeHours: 0 (Firecrawl docs, Exa docs, checked November 10, 2026).

Is WebBaseLoader good enough for production?

For a few cooperative sites, yes. It fetches with a plain HTTP client from your server, so protected sites return challenge pages or errors, and a challenge page with HTTP 200 gets embedded as if it were content. Check each response for text from the real page before you embed it; our guide to blocks that return 200 shows how.

Does Firecrawl return cached pages?

Yes, by default. As of November 2026, Firecrawl returns a cached copy when it has one newer than maxAge, which defaults to 2 days, and a cached result still costs 1 credit. Set maxAge: 0 to always scrape fresh (Firecrawl docs, checked November 10, 2026).

How does Exa decide between cached and live content?

With maxAgeHours. As of November 2026, 0 returns content fetched within the last 60 seconds, -1 always uses the cache, and a value up to 720 uses cached copies younger than that. Each URL's status says whether it was cached or crawled (Exa contents docs, checked November 10, 2026).

How much does it cost to refresh 10,000 pages a day?

About 300,000 fetches a month. As of November 2026, that is $399 on Firecrawl's Growth plan and $300 on Exa Contents for one content type (Firecrawl, Exa, checked November 10, 2026). On String it is $48 of usage at the Growth rate, as of October 2026 (pricing).

How do I avoid re-embedding pages that did not change?

Store a hash of each page's cleaned text, and re-embed only when the hash changes. Delete a page's chunks when the site returns 404 or 410. Firecrawl's change tracking gives you the same signal as a label; with any other fetch API, the hash does it in a few lines.

Which sites are hardest to keep in a RAG index?

Sites behind bot protection, and sites that render content with JavaScript. In the October 30, 2026 run of String's open benchmark, the 17 APIs tested averaged 76.5% on GitHub, Stack Overflow and arXiv. Test your own source list before you commit to a loader.

Sources

• LangChain, Document loader integrations, checked November 10, 2026: docs.langchain.com; web_base reference: reference.langchain.com
• Firecrawl, Change tracking and Faster scraping docs, checked November 10, 2026: change tracking, maxAge; pricing, checked November 9, 2026: firecrawl.dev/pricing
• Exa, pricing and Contents API reference, checked November 10, 2026: exa.ai/pricing, exa.ai/docs/reference/get-contents
• Web Data Frontier Benchmark, October 30, 2026 run: usestring.ai/benchmark and GitHub
• String pricing, as of October 2026: usestring.ai/pricing; fetch docs: portal.usestring.ai/docs/fetch/overview