How to scrape Google search results in 2026

Last updated: September 30, 2026. Benchmark figures come from the September 16, 2026 run of the Web Data Frontier Benchmark. Every script on this page was run on September 30, 2026, and the output shown is what it printed.

To scrape Google search results in 2026, you need a client that runs JavaScript, or a service that runs it for you and hands back the parsed page. A plain HTTP request no longer gets results. On September 30, 2026, a Python requests call to https://www.google.com/search?q=openai from a home connection got HTTP 200 and a 91 KB page with no results in it, only an enablejs redirect. A Chrome User-Agent got the same page. Through the String Web Access API, the same URL came back as parsed JSON: organic results with position, title, URL and snippet, plus People Also Ask, related searches and the AI Overview when Google showed them. The Python below checked three keywords in 14.9 seconds, and the TypeScript returned the top 30 results for one query in 7.3 seconds.

This page is part of our series on the best web scraping APIs, tested.

TL;DR

• Plain requests to Google stopped getting results in January 2025, per developer reports. On September 30, 2026, a plain request got an HTTP 200 page with no results and an enablejs redirect. It is not an error code, so check for results, not the status.
• In the September 16, 2026 benchmark, 6 of 16 scraping APIs returned Google search on all five attempts. Six returned it on none. Google ranked 22nd of 100 sites by difficulty, among the harder quarter of the run.
• String returned Google on all five attempts, in 3.5 seconds on average.
• String's fetch endpoint returns a Google results URL as parsed JSON, not HTML. For more than one page of results, use String's search endpoint: 30 results came back in 4 pages in one call.
• Google's own Custom Search JSON API is closed to new customers, and existing customers have until January 1, 2027 to move, per Google's developer page.

What people who scrape Google run into

Most Google scrapers broke in the same week. On January 16, 2025, users of the googlesearch Python library opened an issue titled "Empty result". One wrote: "Date it stopped working: 1/16/2025" (googlesearch issue #95, January 16, 2025). Libraries that kept working did so by sending the User-Agent of a text browser that Google still served without JavaScript. That stopped in September 2025. In a new "Results is empty" thread, a contributor posted what Google now sends to that browser: "Your browser isn't supported anymore." (googlesearch issue #113, September 10, 2025).

The official route is closing too. Google's Custom Search JSON API, the supported way to get results as JSON, says it "is closed to new customers" (Google for Developers). It also never returned the page a user sees: no ads, no AI Overview, no People Also Ask. For rank tracking, which is what most people scrape Google for, those parts of the page matter.

Why a plain request to Google fails

We sent the benchmark's Google URL, https://www.google.com/search?q=openai, on September 30, 2026, from a home internet connection outside the US.

The page Google sends to a client without JavaScript is not a block page in the usual sense. It has a 200 status and Google's own title. It has no <h3> result titles in it, and it tells the browser to reload with JavaScript on. A scraper that parses it finds zero results and moves on, which is how a rank tracker ends up reporting that every keyword dropped off the page on the same day.

There are two ways past it. Run a real browser, with Playwright or Selenium, and parse the rendered page; that works and it is slow, and Google changes its markup often. Or send the query to a service that renders and parses the page for you.

Ways to get Google search results, compared

What 16 scraping APIs did with Google search

The Web Data Frontier Benchmark sends the same 100 URLs to 16 scraping APIs, five attempts each. The Google URL is the results page for the query "openai". The harness and targets are open source, and so is the raw result file for the September 16 run.

Six of 16 APIs returned Google on all five attempts: 41 passes out of 80 requests. The average API passed 51.3% of attempts on Google, against 66.6% across all 100 sites. Google ranked 22nd of 100 by difficulty. This row splits the field more than most. Scrapfly and ScrapingBee, second and seventh overall, passed none, while Scrapingdog and Browserbase, near the bottom overall, passed all five.

Read this row with one caveat. The benchmark counts a pass when the response is a 2xx and the body contains "OpenAI", compared without regard to case (validateResponse in the harness's check.ts). The enablejs page we got on a plain request contains "openai", because it repeats the query. So a 2xx page with no results in it can pass. The APIs that passed none did not get that far: their attempts did not come back as a 2xx page with the query in it. For your own tests, count the organic results.

How to track Google rankings with Python

This is the job most people scrape Google for: take a list of keywords and find where a domain ranks. Install requests, set your key, and pass a domain and your keywords. Python 3.10 or later.

"""Track Google rankings for a list of keywords with the String Web Access API.

    pip install requests
    export STRING_API_KEY=...
    python google_serp.py usestring.ai "web scraping api" "scraping api python" "serp api"
"""
import os
import sys
from urllib.parse import quote_plus, urlparse

import requests

API = "https://request.usestring.ai/v1/fetch"


def serp(query: str) -> dict:
    url = f"https://www.google.com/search?q={quote_plus(query)}&hl=en&gl=us"
    response = requests.post(
        API,
        headers={"Authorization": f"Bearer {os.environ['STRING_API_KEY']}"},
        json={"url": url, "countryCode": "US"},
        timeout=90,
    )
    response.raise_for_status()
    page = response.json()
    # For a Google results URL, String answers with the parsed page, not the HTML.
    if "results" not in page:
        raise RuntimeError(f"no parsed results for {url}: {str(page)[:200]}")
    return page


def rank_of(domain: str, results: list[dict]) -> int | None:
    for r in results:
        host = urlparse(r.get("url", "")).netloc.lower()
        if host == domain or host.endswith("." + domain):
            return r["position"]
    return None


if __name__ == "__main__":
    domain, keywords = sys.argv[1], sys.argv[2:] or ["web scraping api"]
    for kw in keywords:
        page = serp(kw)
        results = page["results"]
        extras = [k for k in ("overviews", "peopleAlsoAsk", "ads", "places", "videos") if page.get(k)]
        top = urlparse(results[0]["url"]).netloc if results else "none"
        print(f'{kw!r}: {len(results)} organic, #1 {top}, {domain} at {rank_of(domain, results) or "not on page 1"}'
              f' | also on page: {", ".join(extras) or "none"}')
        for q in page.get("peopleAlsoAsk", [])[:2]:
            print(f"    people also ask: {q['question']}")

What it printed on September 30, 2026, in 14.9 seconds for three keywords:

'web scraping api': 9 organic, #1 www.scrapingbee.com, usestring.ai at not on page 1 | also on page: overviews, peopleAlsoAsk
    people also ask: Is web scraping illegal?
    people also ask: Does web scraping require API?
'scraping api python': 8 organic, #1 www.scraperapi.com, usestring.ai at not on page 1 | also on page: overviews, peopleAlsoAsk, videos
    people also ask: Is web scraping illegal?
    people also ask: Is scraper API free?
'serp api': 5 organic, #1 serpapi.com, usestring.ai at not on page 1 | also on page: overviews, peopleAlsoAsk
    people also ask: What does SerpApi stand for?
    people also ask: Is SerpApi illegal?

Yes, that is our own domain off page one for all three. We left it in because it is what the run showed. Note the organic counts: 9, 8 and 5, not 10. When Google puts an AI Overview, videos or People Also Ask on the page, it shows fewer blue links, and a rank tracker that assumes ten per page will be wrong.

Each entry in results carries position, title, url, snippet, displayUrl and source. The parsed Google response from the fetch endpoint is not in the fetch reference; the search reference documents the same fields. Do not page this script with Google's start= parameter. We tried it twice. The first time, page two repeated 2 of page one's results. The second time, page two was page one again: 9 results, 0 new. For more than one page, use the search endpoint in the next section.

How to get the top 30 Google results with TypeScript

No dependencies. Node 22.6 or later runs TypeScript directly (we ran it on Node 24), and so does Bun. This one calls String's documented search endpoint, which pages Google for you until it has the number of results you asked for.

// Pull the top 30 Google results for one query with the String search endpoint.
//   export STRING_API_KEY=...
//   node google_search.ts "web scraping api"      (Node 22.6+; or: bun google_search.ts "...")

type Result = { position: number; title: string; url: string; snippet?: string };
type Serp = {
  results: Result[];
  zeroResults: boolean;
  peopleAlsoAsk?: { question: string }[];
  relatedSearches?: string[];
  overviews?: { text: string }[];
  paging?: Record<string, unknown>;
};

const query = process.argv[2] ?? "web scraping api";
const response = await fetch("https://request.usestring.ai/v1/search", {
  method: "POST",
  headers: { Authorization: `Bearer ${process.env.STRING_API_KEY}`, "Content-Type": "application/json" },
  body: JSON.stringify({ query, engine: "google", country: "US", language: "en", searchCount: 30 }),
});
if (!response.ok) throw new Error(`String returned ${response.status}: ${await response.text()}`);
const serp = (await response.json()) as Serp;
if (serp.zeroResults) throw new Error(`Google returned no results for "${query}"`);

// In one of our runs a Reddit result came back with an empty url. Keep such rows out of the ranks.
const noUrl = serp.results.filter((r) => !r.url);
serp.results = serp.results.filter((r) => r.url);
const hosts = serp.results.map((r) => new URL(r.url).hostname.replace(/^www\./, ""));
console.log(`"${query}": ${serp.results.length} organic results, ${new Set(serp.results.map((r) => r.url)).size} unique URLs, ${new Set(hosts).size} domains`);
console.log(`paging: ${JSON.stringify(serp.paging ?? null)} | results without a url: ${noUrl.length}${noUrl[0] ? ` ("${noUrl[0].title.slice(0, 40)}")` : ""}`);
for (const r of serp.results.slice(0, 3)) console.log(`${String(r.position).padStart(2)}. ${new URL(r.url).hostname}  ${r.title.slice(0, 50)}`);
const last = serp.results.at(-1);
if (last) console.log(`...\n${String(last.position).padStart(2)}. ${new URL(last.url).hostname}  ${last.title.slice(0, 50)}`);
console.log(`AI Overview on the page: ${serp.overviews?.length ? "yes" : "no"} | people also ask: ${serp.peopleAlsoAsk?.length ?? 0} | related searches: ${serp.relatedSearches?.length ?? 0}`);

We ran it on September 30, 2026, on Node 24 and again on Bun, with the same output. It returned in 7.3 seconds:

"web scraping api": 30 organic results, 30 unique URLs, 27 domains
paging: {"pages":4,"complete":true} | results without a url: 0
 1. www.scrapingbee.com  ScrapingBee – The Best Web Scraping API
 2. www.firecrawl.dev  Firecrawl - The web data API to search, scrape, an
 3. www.scraperapi.com  ScraperAPI: Scale Data Collection with a Simple We
...
30. www.g2.com  How do I scrape API from my website?
AI Overview on the page: yes | people also ask: 4 | related searches: 8

Thirty results took four Google pages, which is the point above about fewer than ten per page. The first run of this script crashed on a result with an empty url field, and a Reddit thread result in one of the Python runs had an empty url too. The filter in the script is the fix. Keep it.

The failure mode: a 200 page with no results

Google's failures do not come with error codes, which makes them easy to miss.

• The enablejs page. HTTP 200, about 91 KB, Google's own title, zero results. What any client without JavaScript gets.
• The "unusual traffic" page. Google's CAPTCHA interstitial for traffic it scores as automated. We did not hit it in this run. It is what people report at volume from one IP.
• A real page for the wrong place. Google localizes results by the IP the request comes from. A US keyword checked from a German exit is a different results page, with a 200 and a full set of results.

On the last one: we asked for a US exit on every request in this run, and we checked. Four "weather" queries returned weather.com, weather.gov and Wunderground in the top three, and three "pizza delivery" queries with no gl parameter returned places and delivery pages in the Bronx, New York. Seven of seven came back from the US. On a different day in the same week, some String requests with a US exit came back localized to another country on another site. A US exit does not guarantee US results. Check a location signal in the page on every run, such as a local pack's addresses, and refetch any page that comes back from the wrong place.

Three checks catch all of it:

• Count the organic results. Zero from a query that had results yesterday is a block, not a ranking change.
• Check the location. Add gl=us&hl=en to the URL, and read an address or a country-specific domain from the page.
• Drop rows with no URL. A rank computed from a list with a blank row in it is off by one from that row down.

What it costs

The Google responses in our runs carried no billing header, so we cannot show you what each one billed as from our own logs. String's search reference says each Google results page that answered is billed as one search, and a 100-result request typically costs 11 to 13 searches. Failed pages are not billed. The public pricing page does not list a per-search rate, so check your plan in the portal before you size a rank-tracking job.

In search units, from the runs above: the Python script cost one search per keyword, three for three keywords. The TypeScript script's 30 results took four pages, so four searches.

FAQ

How to scrape Google search results?

Send the query to a client or service that runs JavaScript. In our test on September 30, 2026, Google served a plain request an enablejs page with no results. With String, post https://www.google.com/search?q=<query>&hl=en&gl=us to the fetch endpoint and read results, or post the query to the search endpoint with searchCount for more than one page.

How do I scrape Google search results with Python?

Send the Google URL to a fetch API with requests and read the parsed JSON. The Python script on this page tracks where a domain ranks for a list of keywords, and it checked three keywords in 14.9 seconds on September 30, 2026. Parsing Google's HTML with BeautifulSoup no longer works on a plain request, because the page has no results in it.

Why does my Google scraper return no results?

Google wants JavaScript. In our test, a plain request got HTTP 200 with a page that tells the browser to enable JavaScript, and it had no results in it. In our test on September 30, 2026, a Chrome User-Agent did not change that. The text-browser User-Agent trick that some libraries used stopped working in September 2025.

Is Google's Custom Search API still available?

Not for new projects. Google's developer page says the Custom Search JSON API "is closed to new customers", and existing customers have until January 1, 2027 to move to another solution. It also left out ads, the AI Overview and People Also Ask, which a rank tracker usually needs.

How many results does Google show per page?

Fewer than ten, often. In our run on September 30, 2026, three keywords returned 9, 8 and 5 organic results on page one, because the AI Overview, videos and People Also Ask took space. Getting 30 results took four pages.

Can I scrape Google AI Overviews?

Yes, when Google shows one. String's parsed Google response puts it in overviews as text, and all three of our test keywords had one. For Google's AI Mode, the search endpoint takes engine: "google_ai_mode" and returns a structured aiMode answer; we did not run it for this page.

How do I scrape Google results for a specific country?

Add gl=<country>&hl=<language> to the URL and set the exit country on the request, countryCode on String's fetch endpoint or country on its search endpoint. Then check a location signal in the page. A matching exit makes the right country likely, not certain.

Sources

• String benchmark: Web Data Frontier Benchmark, harness and targets, September 16, 2026 raw results, Google Search target definition
• String docs: Fetch API reference, Search API reference, pricing
• Google: Custom Search JSON API overview
• Practitioner reports: googlesearch issue #95, January 2025, googlesearch issue #113, September 2025
• Related on usestring.ai: best web scraping APIs, how to scrape Reddit, how to scrape X (Twitter), how to scrape Amazon product data, how to scrape Zillow, the Web Access API