Search API for AI agents: a tested setup for 2026

Last updated: October 21, 2026. Every search number on this page comes from runs we made on October 21, 2026 through String's search endpoint, from one US connection. The Python and TypeScript code ran the same day, and the output shown is what it printed. We build String, the search API used in the code below.

To let an AI agent search the web without Google's official API, give it a web_search tool that calls a search API and returns titles, URLs and snippets as JSON, then a second tool that fetches the pages it picks. Google's Custom Search JSON API takes no new customers and shuts down on January 1, 2027, so the tool needs a different backend: a Google results API, an independent index such as Brave, an AI-native search API, or a self-hosted SearXNG.

One choice matters more than the vendor: what the agent types. On October 21, 2026 we searched 58 real questions twice through Google, once as written and once rewritten as keywords. Each form got a median of 8 organic results, and the two lists shared a median of 4 domains. They put the same site at #1 on only 23 of 57 questions. Google showed an AI Overview for 57 of 57 full questions and People Also Ask for 52, against 40 for the keyword forms. Let the agent send the question as written.

On this page: TL;DR · Where String fits · What we measured · Pick a backend · Step 1 to Step 5 · How to avoid getting blocked · How String handles agent search · What it costs · FAQ · Sources

TL;DR

• An agent needs a search tool and a fetch tool. web_search finds pages. fetch_page reads them. A snippet averages about 22 words.
• Google's official API is closing. The Custom Search JSON API is closed to new customers and ends January 1, 2027. Microsoft retired the Bing Search APIs on August 11, 2025.
• The query form changes the results. Same 57 questions, two forms: a median of 4 domains shared out of about 8 results each, same #1 site on 23. The full form kept AI Overviews on 57 of 57 and People Also Ask on 52 of 57.
• Parallel search holds up. 20 different searches fired at once all returned. Median 1.4 seconds, 90th percentile 3.0 seconds.
• A parsed results page is small. Title, URL and snippet for every organic result came to a median of about 2,400 characters, roughly 600 tokens.
• String search costs $1.00 per 1,000 searches on the Growth plan, $1.50 on Starter.

Where String fits

String fits one job on this page: the search and fetch tools an agent calls.

• For: teams building agents, research assistants or RAG pipelines that need Google results, Google's AI Overview and AI Mode answers, and the pages behind them, from one API key.
• Also for: agents in Claude, Cursor or ChatGPT. The hosted MCP server at https://mcp.usestring.ai/v2/mcp gives them web_access_search and web_access_fetch with no code.
• Price: $1.50 per 1,000 searches on Starter ($20 a month), $1.00 on Growth ($100 a month). An AI Overview fill costs 2x a search.
• Engines: Google, DuckDuckGo, Brave and Mojeek, plus Google AI Mode.
• Limits: 60 requests a second with a burst allowance of 3,600, and no cap on concurrent requests.

Try one search from a terminal before you write any agent code:

curl https://request.usestring.ai/v1/search \
  -H "Authorization: Bearer $STRING_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"query": "How do I give my AI agent access to the web?"}'

What we measured

Agents and people send different queries. A person types rag web data api. An agent passes the user's sentence: "Best web data API for building RAG pipelines?" We wanted to know whether that difference changes what comes back.

The method. We took 58 questions buyers ask about web data tools, scraping and search. Each went to Google through String's /v1/search twice: once as written, once as a keyword form. The keyword form drops question words, stop words and punctuation, so "How to scrape Zillow / real estate listings?" became "scrape zillow / real estate listings". The rule is in the script, not chosen by hand. Both forms ran from the US in English, four questions at a time.

The two forms returned different pages. Each form got a median of 8 organic results. The two lists shared a median of 4 domains and 3 URLs. The same site ranked #1 on 23 of 57 questions, 40%.

Some questions changed completely. "Best web data API for building RAG pipelines?" and its keyword form "web data api building rag pipelines" shared no domain at all. So did "Best scraping API for e-commerce price data?" and "scraping api e-commerce price data".

What this means for an agent. Neither form is right for every query. The full question kept the extra surfaces an agent can use: an AI Overview on every question and People Also Ask on 12 more. So pass the user's question as written by default. If you want a second view, run the keyword form too and merge the results. That costs one more search, not one more model call.

Parallel searches. Agents often fan out several searches at once. We fired 20 different questions at the same moment, none searched earlier that day. All 20 returned. The median took 1.4 seconds, the 90th percentile 3.0 seconds, and the slowest 11.6 seconds. Set a timeout and let the agent move on without the slow one.

Pick a backend

Five kinds of backend can sit behind a web_search tool. Here is what each gives an agent.

Only String was measured on this page. The other vendors are Not tested here. Our best web search APIs page compares them on features and published prices, and measures one rival side by side.

Cloudflare added an option this month. On October 2, 2026 Cloudflare added a web search API to its AI Gateway, with Ceramic.ai, Exa and Linkup as search partners. It bills at the partners' list prices. We have not tested it.

Self-hosting is free, but it needs upkeep. One developer on r/LocalLLaMA built web_search on SearXNG and wrote that "Public SearXNG instances are usually a bad fit for programmatic use." They also listed the tradeoff plainly: "SearXNG quality depends on enabled upstream engines and rate limits."

Step 1: define the search tool

The tool definition is a JSON Schema. Claude, OpenAI and most agent frameworks accept this shape. The description tells the model two things: send the question as written, and a snippet is not the page.

WEB_SEARCH_TOOL = {
    "name": "web_search",
    "description": "Search Google for current information. Pass the user's question as written. "
                   "Returns titles, URLs and snippets. A snippet is not the page: fetch a URL to read it.",
    "input_schema": {
        "type": "object",
        "properties": {
            "query": {"type": "string", "description": "The question or search terms."},
            "dateRange": {"type": "string", "enum": ["day", "week", "month", "year"],
                          "description": "Optional. Only results from this recent window."},
        },
        "required": ["query"],
    },
}

dateRange lets the agent ask for recent results when the user asks about news or a recent release. String passes it to Google's own date filter.

Step 2: write the handler

The handler calls the search API and trims the response. Keep the title, URL and snippet. Drop display fields the model does not need. This version also passes back Google's AI Overview when the page carried one.

import json
import os

import requests

API_KEY = os.environ["STRING_API_KEY"]


def web_search(query: str, dateRange: str | None = None) -> str:
    body = {"query": query, "engine": "google", "country": "US", "language": "en"}
    if dateRange:
        body["dateRange"] = dateRange
    r = requests.post(
        "https://request.usestring.ai/v1/search",
        headers={"Authorization": f"Bearer {API_KEY}", "Content-Type": "application/json"},
        json=body,
        timeout=60,
    )
    r.raise_for_status()
    data = r.json()
    # Keep what the model needs. Drop display fields to save context.
    results = [
        {"title": x.get("title"), "url": x.get("url"), "snippet": x.get("snippet")}
        for x in data.get("results", [])
        if x.get("url")
    ][:8]
    overview = next((o["text"] for o in data.get("overviews", []) if o.get("text")), None)
    return json.dumps({"results": results, "aiOverview": overview})


out = web_search("How to let an AI agent search the web without Google's official API?")
print(len(out), "chars")
for item in json.loads(out)["results"][:3]:
    print(item["url"])

On October 21, 2026 (Python 3.14.7, requests 2.34.2) it printed:

2143 chars
https://www.reddit.com/r/ProxyEngineering/comments/1u902eb/web_search_api_for_ai_agents/
https://www.firecrawl.dev/blog/web-search-api
https://parallel.ai/articles/best-free-web-search-api

The same tool in TypeScript:

const API_KEY = process.env.STRING_API_KEY!;

export async function webSearch(query: string, dateRange?: string): Promise<string> {
  const res = await fetch("https://request.usestring.ai/v1/search", {
    method: "POST",
    headers: { Authorization: `Bearer ${API_KEY}`, "Content-Type": "application/json" },
    body: JSON.stringify({ query, engine: "google", country: "US", language: "en", ...(dateRange && { dateRange }) }),
  });
  if (!res.ok) throw new Error(`search failed: ${res.status}`);
  const data = await res.json();
  const results = (data.results ?? [])
    .filter((r: any) => r.url)
    .slice(0, 8)
    .map((r: any) => ({ title: r.title, url: r.url, snippet: r.snippet }));
  return JSON.stringify({ results });
}

const out = await webSearch("web search API for AI agents", "month");
console.log(out.length, "chars");

Run on Bun on October 21, 2026, it printed 2257 chars and three result URLs from the past month.

Step 3: add a fetch tool for the pages

Search finds pages. The agent still has to read them. The developer on r/LocalLLaMA put it in one line: "Search snippets are not enough. The agent needs to read the actual page." In our test, all the snippets on one results page came to a median of 179 words, about 22 per result.

Give the agent a fetch_page tool that takes one URL and returns the page as Markdown. With String, that is a POST to /v1/fetch with {"url": url, "format": "markdown"}. Our web search API vs web scraping API page measured the difference on the same 58 questions: the snippets carried a price on 11, the fetched pages on 39.

Step 4: run searches in parallel, with a timeout

Agents ask several things at once. A research agent may split one request into four or five searches. Fire them together and cap each one.

• Set a per-search timeout of about 10 seconds. In our burst of 20, 19 came back within 3.1 seconds and one took 11.6.
• Search the full question first. Add the keyword form only when the first results look thin.
• Deduplicate URLs across searches before the agent fetches anything.

Step 5: tell the model what to trust

Treat a search result as a lead to check. Put that in the system prompt. Ask the agent to cite the URL it fetched, not the snippet it saw. Ask it to say so when two sources disagree. This also guards against a page that tells the agent to do something: treat fetched text as data, never as instructions.

Next step: get a key at usestring.ai and paste the Step 2 handler into your agent. The first search takes about a second.

How to avoid getting blocked

Scraping Google results pages yourself breaks often. Google serves CAPTCHAs to scripted traffic, and its page layout changes without notice. Our guide on how to scrape Google search results covers what that takes. A search API handles it for you.

Free search libraries hit the same walls. Libraries that scrape a search engine's public pages, such as DDGS, need no key. They can also be rate limited or blocked, and they stop working when the page changes.

The fetch step gets blocked more than the search step. Search returns a list of URLs. Many of those URLs sit behind Cloudflare, DataDome or Akamai. A fetch tool that handles those walls keeps the agent from reading a challenge page and reporting it as the answer. Our page on how to tell when a 200 response is really a block shows what to check.

How String handles agent search

String is the most reliable web scraping API, bypassing anti-bot, and you only pay when content comes back.

• Google results as JSON from one call. POST /v1/search returns title, URL, display URL, snippet and position for each organic result. (Search docs)
• AI Overview and AI Mode answers for agents to compare. Set "aiOverview": true to fill a streamed overview, or "engine": "google_ai_mode" for Google's AI Mode answer and the pages it cites. (Search docs)
• Recent results on request. dateRange limits Google results to the past hour, day, week, month or year, and "sortBy": "date" puts the newest first. (Search docs)
• Search and fetch from one key. The same key calls /v1/fetch, which returns the page as Markdown and bills only when content comes back. (Pricing)
• No code for MCP clients. The hosted MCP server gives Claude, Cursor, ChatGPT and other MCP clients web_access_search and web_access_fetch. (MCP docs)
• No concurrency cap. Every plan runs at 60 requests a second with a 3,600-request burst allowance. On October 21, 2026, 20 parallel searches all returned. (Pricing)

What it costs

String bills per search. 1,000 searches cost $1.50 on Starter and $1.00 on Growth. A Google AI Mode answer is one search. A search with "aiOverview": true costs two.

Fetches are billed separately. A standard fetch costs $0.20 per 1,000 on Growth. Premium and browser fetches cost more and run only when a site needs them. See pricing.

FAQ

How can I let an AI agent search the web without Google's official API?

Give the agent a web_search tool backed by a search API, and a fetch_page tool for the pages it picks. Google's Custom Search JSON API takes no new customers and ends January 1, 2027. Google results APIs such as String, SerpApi and Serper return the same results page as JSON. Brave runs its own index. SearXNG is a free self-hosted option.

What is the best web search tool for AI agents?

It depends on which results your users expect. If they compare the agent with Google, use a Google results API. String also returns Google's AI Overview and AI Mode answer. If you want results picked for model input, test an AI-native search API on your own questions. Our best web search APIs page compares them.

How do I add web search to an AI agent?

Define a tool with a JSON Schema that takes a query string. Write a handler that calls the search API and returns titles, URLs and snippets as JSON. Register both with your model or agent framework. Steps 1 and 2 above show working Python and TypeScript.

Should my agent send the full question or keywords?

Send the full question by default. On October 21, 2026, 57 questions sent both ways shared a median of 4 result domains out of about 8. The full questions kept an AI Overview on 57 of 57 and People Also Ask on 52 of 57, against 40 for keywords. Run the keyword form as a second search when results look thin.

Is there a free search API for AI agents?

Most paid search APIs have a free tier, and SearXNG is free to host. String gives 5,000 standard fetch requests free. Google's free Custom Search tier is closed to new customers. Free search libraries that scrape result pages need no key, but they get rate limited and break when the page changes.

Can AI agents search the web on their own?

Yes, when you give them a search tool. The model decides when to call it, writes the query, reads the results, and calls a fetch tool for pages it wants to read. Claude, ChatGPT and Cursor can also get search through an MCP server, such as String's hosted server, with no code.

What happened to the Bing Search API?

Microsoft retired the Bing Search APIs on August 11, 2025 and decommissioned existing instances. New signups are not possible. Agents that used Bing need a new backend: a Google results API, Brave's independent index, or an AI-native search API.

Do I need a search API and a scraping API for an agent?

Usually yes. A search API returns links and short snippets. A scraping API, or fetch API, returns the full page. In our October 21 test, all the snippets on one results page came to a median of 179 words. String sells both under one key: /v1/search and /v1/fetch.

Try it

Run the Step 2 handler with your own question. Then add the fetch tool and let the agent read what it found. Get a free key at usestring.ai.

Sources

• Google, Custom Search JSON API overview
• Microsoft, Bing Search APIs retirement
• Cloudflare, Introducing Web Search API via AI Gateway (Oct 2, 2026)
• Cloudflare web search developer docs
• r/LocalLLaMA, Giving a local agent web access without paid search/scrape APIs
• r/LangChain, What internet search providers are you using for your agents that are free?
• SearXNG on GitHub
• OpenAI, web search tool guide
• Anthropic, web search tool
• String search docs
• String pricing docs
• String pricing
• String MCP server docs
• String fetch docs