How to Handle Google /goto Redirect URLs and CAES Tokens in a SERP Scraper

• Treat /goto?url=CAES... and /url?... as redirect layers, not destinations.
• Do not decrypt CAES. Resolve with GET and allow_redirects=False, then read Location.
• Prefer Path A: WebScrapingAPI SERP API structured JSON when organic link fields are destinations.
• Use Path B: Scraper API or Browser API HTML only when you need raw markup—then classify → dedupe → resolve.
• HTTP 200 is not a usable SERP. Validate result containers before parsing.
• Deduplicate identical /goto tokens within one page; reuse connections; bound concurrency.
• Store a request fingerprint (query, gl, hl, device, domain, time) with every observation.
• Google’s link format is not a permanent contract. Design for direct URLs, /goto, and /url.

Historically, scrapers often read the destination straight from the organic anchor. Today the same conceptual result can appear in several shapes:

/goto is a Google-hosted redirect endpoint. The url query value often starts with CAES—not because “CAES” is a product name, but because that is how a common binary envelope looks after Base64URL encoding. Opening the link asks Google for a redirect to the real destination.

/url is an older (and still seen) Google redirect path. Some clients reportedly receive /url with a CAES payload instead of a plaintext destination. Same scraper rule either way: classify the link, then keep a direct destination or resolve the wrapper. Do not assume opaque tokens only show up on /goto.

Google has described redirect layers as part of technical measures against evolving abuse of Search. Exact rollout logic is not a public contract. Build for multiple formats.

Most production rank-tracking and SERP monitoring jobs need destination URLs + titles + snippets + ranks, not Google’s presentation HTML. Pick the product path first.

Send a contextual Google query to WebScrapingAPI’s SERP endpoint and read parsed collections:

• Endpoint: https://serpapi.webscrapingapi.com/v2
• Required: api_key, engine=google, q
• Common optional controls: device, domain, uule, hl, gl, start, num, tbm, ibp
• Async alternative: engine=google_async, then retrieve via the Snapshot API

Product pages: SERP API · Google Search API
Docs: Access the API · Google Search engine

What the public docs show today: organic result examples use link fields that look like final destination URLs (for example https://www.history.com/ or https://en.wikipedia.org/wiki/History), not google.com/goto wrappers. Field docs describe link as the destination URL.

Check the link field on your own live response:

• Before you rely on Path A alone, inspect organic link values from a dated SERP call on your account.
• Public docs show destination-looking link examples; whether your live production responses wrap any results in /goto is something only your own sample can tell you—do not assume either way from this article.

Also note a documentation schema dual naming: some pages show an organic array (link, display_link, rank), while the organic-results doc shows organic_results (link, displayed_link, position). Both document link as the destination. Parse defensively for either shape and validate against a live response.

Use HTML when you need custom selectors, layout research, or a workflow the structured SERP engine does not cover:

• Scraper API: https://api.webscrapingapi.com/v2 with api_key + url
• Set render_js=1 when the target needs client-side JavaScript before you receive the body
• Browser API: when you need documented waits, interactions, or browser-state controls

Validate a usable SERP. An HTTP 200 can still be an interstitial, consent wall, CAPTCHA, or “enable JavaScript” shell. Before extracting links, assert that recognizable result containers exist (for example anchors under #rso / #search, or equivalent markers you maintain). If validation fails, retry with rendering, adjust locale/device context, or fall back to Path A—do not silently store empty ranks.

After you extract /goto URLs from HTML, resolve them with plain HTTP (see below). You generally do not need another rendered scrape credit per organic link.

Resolution is the core of Path B—and a safety net if any wrapper ever appears in structured output.

import requests

def resolve_google_goto(url: str, timeout: float = 10.0) -> str:
    """Resolve a Google /goto (or similar) redirect without following to the destination site."""
    response = requests.get(
        url,
        allow_redirects=False,
        timeout=timeout,
        headers={"User-Agent": "Mozilla/5.0"},
    )
    if response.status_code != 302:
        raise RuntimeError(f"Expected 302, got {response.status_code} for {url[:80]}...")
    location = response.headers.get("Location")
    if not location:
        raise RuntimeError("Redirect response has no Location header")
    return location  # keep full value, including #:~:text= fragments when present

Rules that matter in production:

1. allow_redirects=False — you want Google’s Location, not a second hop to the publisher.
2. Prefer GET over HEAD. Public write-ups report that HEAD can return 200 without a Location header while GET returns 302 with the destination. Treat that as industry guidance; confirm HEAD vs GET behavior on a live /goto token from your own pipeline before you hard-code the method.
3. Preserve Location intact. Google may return text fragments such as #:~:text=.... Strip them only if your product explicitly needs canonical URLs.
4. Resolve outside the scrape API when the token is portable—collect via Scraper/Browser HTML, resolve with a normal client session.

Minimal cURL pattern:

curl -sS -o /dev/null -D - \
  "https://www.google.com/goto?url=CAES..." \
  | grep -i '^location:'

Do not use -L if you only need the redirect target.

Never build the parser around the substring CAES alone. Classify the absolute URL first:

from urllib.parse import parse_qs, urljoin, urlparse

GOOGLE_BASE_URL = "https://www.google.com/"

def is_google_host(host: str) -> bool:
    host = (host or "").lower()
    return host == "google.com" or host.endswith(".google.com")

def classify_google_link(href: str) -> tuple[str, str]:
    absolute = urljoin(GOOGLE_BASE_URL, href)
    parsed = urlparse(absolute)
    host = parsed.hostname or ""

    if is_google_host(host) and parsed.path == "/goto":
        token = parse_qs(parsed.query).get("url", [""])[0]
        if token.startswith("CAES"):
            return "goto", absolute
        return "goto", absolute  # still a /goto wrapper even if token shape changes

    if is_google_host(host) and parsed.path == "/url":
        return "google_url", absolute

    if parsed.scheme in {"http", "https"} and not is_google_host(host):
        return "direct", absolute

    return "other", absolute

Suggested handling:

HTML → URL pipeline:

SERP fetch (SERP API JSON or Scraper/Browser HTML)
        ↓
validate usable payload (JSON modules or SERP DOM)
        ↓
extract candidate hrefs / link fields
        ↓
classify → direct | goto | google_url | other
        ↓
dedupe wrappers (same token often repeats in one SERP)
        ↓
resolve with GET, bounded concurrency, connection reuse
        ↓
merge destinations + store request fingerprint + outcome codes

One SERP page often repeats the same /goto token across title and sitelink anchors. Resolving every anchor wastes requests. Preserve order while uniquing:

unique_goto_urls = list(dict.fromkeys(goto_urls))

Use a small worker pool and one requests.Session per worker so TCP/TLS can be reused. Exact worker counts are environment-specific—tune against your latency and error rates rather than copying another vendor’s benchmark numbers.

Comparable SERP data needs context beside the URLs:

• q, gl, hl, device, domain, uule (if used), start, num, tbm/ibp
• Observation timestamp and schema version
• Fetch path (serp_api vs scraper_html vs browser_html)
• Counts: direct / goto / google_url / resolved / failed

That aligns with WebScrapingAPI’s responsible web data guidance: preserve observation context so downstream teams can interpret each record.

Short answer: treat it as conditional, not as a fixed rule.

Third-party write-ups (including ScrapingBee’s September 2026 deep dive) have reported /goto in manually operated Chrome and Edge, in Playwright sessions, and in HTML fetched through scraping APIs, while some privacy-oriented browsers still showed direct destinations and Safari sometimes showed /url with CAES. Those are competitor-reported observations, not WebScrapingAPI re-runs, and they are not a Google contract. IP or country changes alone were reported as insufficient to force direct links in those tests—again, qualify and do not overfit.

Safe production assumptions:

• Automation is not the only trigger.
• Browser engine alone is not a reliable switch.
• Geolocation alone is not a reliable switch.
• Your scraper must accept direct, /goto, and /url over time.

If you need a browser matrix for your own compliance story, run a small internal smoke test and date it—do not publish unverified rules as facts.

You do not need this section to ship a working scraper. It exists so teams stop spending cycles on dead ends.

At a high level, industry analyses describe:

1. Base64URL decoding of the CAES... parameter → binary
2. A compact protobuf-like envelope (commonly a constant field plus a length-delimited blob)
3. A protected payload whose outer bytes resemble a cryptography library output prefix
4. No recoverable plaintext destination for practical scraping purposes

Resolve-first policy: decode for curiosity or malware research if you must; for SERP products, call /goto and read Location. Trying to reverse the protected blob is the wrong problem.

Python:

import os
import requests

api_key = os.environ["WSA_API_KEY"]

response = requests.get(
    "https://serpapi.webscrapingapi.com/v2",
    params={
        "api_key": api_key,
        "engine": "google",
        "q": "best trail shoes",
        "gl": "us",
        "hl": "en",
        "device": "mobile",
    },
    timeout=120,
)
response.raise_for_status()
payload = response.json()

# Docs may use organic or organic_results — support both.
organic = payload.get("organic") or payload.get("organic_results") or []

for row in organic:
    link = row.get("link")
    title = row.get("title")
    # Check link on your live response; if it looks like google.com/goto or /url, classify + resolve.
    print(title, "→", link)

Node.js:

const apiKey = process.env.WSA_API_KEY;
if (!apiKey) throw new Error("WSA_API_KEY is required");

const parameters = new URLSearchParams({
  api_key: apiKey,
  engine: "google",
  q: "best trail shoes",
  gl: "us",
  hl: "en",
  device: "mobile",
});

const response = await fetch(
  "https://serpapi.webscrapingapi.com/v2?" + parameters,
  { signal: AbortSignal.timeout(120_000) }
);
if (!response.ok) {
  throw new Error("Google Search API status " + response.status);
}
const payload = await response.json();
const organic = payload.organic || payload.organic_results || [];
for (const row of organic) {
  console.log(row.title, "→", row.link);
}

cURL:

: "${WSA_API_KEY:?WSA_API_KEY is required}"
curl --get --fail-with-body --max-time 120 \
  "https://serpapi.webscrapingapi.com/v2" \
  --data-urlencode "api_key=$WSA_API_KEY" \
  --data-urlencode "engine=google" \
  --data-urlencode "q=best trail shoes" \
  --data-urlencode "gl=us" \
  --data-urlencode "hl=en" \
  --data-urlencode "device=mobile"

For queued workloads, use engine=google_async and the documented Snapshot retrieval flow instead of blocking on the synchronous engine.

from __future__ import annotations

import os
import threading
from concurrent.futures import ThreadPoolExecutor
from urllib.parse import urlencode

import requests
from bs4 import BeautifulSoup

# reuse classify_google_link + resolve_google_goto from earlier sections

SCRAPER_API = "https://api.webscrapingapi.com/v2"
GOOGLE_SEARCH = "https://www.google.com/search"
_thread_local = threading.local()


def fetch_google_html(query: str, api_key: str) -> str:
    target = GOOGLE_SEARCH + "?" + urlencode({"q": query, "hl": "en"})
    response = requests.get(
        SCRAPER_API,
        params={
            "api_key": api_key,
            "url": target,
            "render_js": "1",  # enable when Google returns a JS shell instead of a SERP
        },
        timeout=120,
    )
    response.raise_for_status()
    return response.text


def assert_usable_serp(html: str) -> None:
    soup = BeautifulSoup(html, "html.parser")
    if not soup.select("#rso a[href], #search a[href]"):
        raise RuntimeError("No recognizable Google search results found (unusable SERP body)")


def extract_links(html: str) -> tuple[list[str], list[str], list[str]]:
    soup = BeautifulSoup(html, "html.parser")
    direct, goto, google_url = [], [], []
    for anchor in soup.select("#rso a[href], #search a[href]"):
        href = anchor.get("href") or ""
        kind, url = classify_google_link(href)
        if kind == "direct":
            direct.append(url)
        elif kind == "goto":
            goto.append(url)
        elif kind == "google_url":
            google_url.append(url)
    return direct, goto, google_url


def get_worker_session() -> requests.Session:
    session = getattr(_thread_local, "session", None)
    if session is None:
        session = requests.Session()
        _thread_local.session = session
    return session


def resolve_goto(url: str) -> tuple[str, str | None]:
    session = get_worker_session()
    try:
        response = session.get(url, allow_redirects=False, timeout=10)
    except requests.RequestException:
        return url, None
    if response.status_code != 302:
        return url, None
    return url, response.headers.get("Location")


def resolve_many(urls: list[str], workers: int = 5) -> dict[str, str]:
    unique_urls = list(dict.fromkeys(urls))
    with ThreadPoolExecutor(max_workers=workers) as executor:
        results = executor.map(resolve_goto, unique_urls)
    return {src: dest for src, dest in results if dest}


def main() -> None:
    api_key = os.environ["WSA_API_KEY"]
    html = fetch_google_html("best trail shoes", api_key)
    assert_usable_serp(html)

    direct, goto, google_url = extract_links(html)
    direct = list(dict.fromkeys(direct))
    goto = list(dict.fromkeys(goto))

    resolved = resolve_many(goto, workers=5)
    # Optionally add handlers for google_url wrappers here.

    destinations = list(dict.fromkeys(direct + list(resolved.values())))
    for url in destinations:
        print(url)

    # Persist fingerprint + counts in your store:
    # path=scraper_html, q, hl, render_js=1, n_direct, n_goto, n_resolved, n_failed


if __name__ == "__main__":
    main()

Install: pip install requests beautifulsoup4. Keep WSA_API_KEY in the environment—never hard-code it.

Public SERP URLs are still web data. Before you scale collection:

1. Name the business purpose (rank monitoring, competitor visibility, research).
2. Define the public-source scope (Google domains, markets, devices, exclusions).
3. Keep a minimum useful record—not every field Google renders.
4. Assign downstream owners for access, retention, and decisions.
5. Preserve observation context so results stay interpretable.

Read WebScrapingAPI’s operating brief: Responsible web data. Your team owns purpose, source scope, and downstream use; WebScrapingAPI operates the selected API path. This article is not legal advice and is not a guide to circumvent security controls—it is a practical URL-handling pipeline for legitimate public-search observation.

No. For scraping and rank tracking, GET the /goto URL with redirects disabled and read Location. Local Base64URL/protobuf decoding does not yield a practical plaintext destination.

Prefer GET. Public reports say HEAD may return 200 without Location while GET returns 302 with the destination. Confirm HEAD vs GET on a live /goto token from your own pipeline before locking in client behavior.

Both are Google redirect endpoints that can sit between the SERP and the destination. /goto commonly carries a CAES... url parameter. /url is an older-style wrapper that may still appear (sometimes with opaque tokens). Classify both; resolve what you cannot parse as a direct destination.

Prefer the Google Search SERP API when structured organic results meet your schema—less HTML fragility, clearer request fingerprint. Use Scraper API or Browser API when you need raw markup or custom interaction. Docs examples show final-looking link values; check the link field on your own live response and record the date before you assume the format under Google’s /goto rollout.

Third-party reports suggest tokens can remain resolvable for more than a day and may work outside the original browser session. Maximum lifetime is unknown. Treat them as opaque temporary redirects, not permanent IDs. (Competitor-reported; not re-run here.)

Not reliably as a sole cause. Competitor-reported tests saw /goto in manual Chrome as well as automated Chromium. Do not bake a “Playwright = /goto, Brave = direct” rule into production logic.

Industry reports describe tokens that could be collected in one environment and resolved later with a plain HTTP client. Useful for separating HTML fetch from redirect resolution—but not a guarantee for every future token variant. Instrument failures.

Google has framed redirect measures as part of protecting Search against evolving abuse. Exact selection logic is not fully public. From a scraper perspective, destinations are no longer always plaintext in HTML.

Treat it as an unusable SERP body. Enable render_js=1 when appropriate, confirm the target URL and locale, or switch to the structured SERP API. Never trust status codes alone.

Store destinations with the request fingerprint (q, gl, hl, device, domain, offsets, fetch path, timestamp). That matches responsible observation practice and prevents false “ranking changes” caused by mixed locales or devices.

Hosted options that return structured search results take the resolver off your plate: our best SERP APIs roundup compares them, and the SERP API quick start guide is the place to begin with ours.

1. Call the synchronous Google engine on https://serpapi.webscrapingapi.com/v2 with api_key, engine=google, and q.
2. Inspect organic link values on a live response from your account and record the date.
3. If you stay on HTML, ship classify → dedupe → resolve before you expand keyword volume.
4. Review responsible web data with your data owners.

Start with the Google Search API, compare SERP API engines, or fetch page HTML via Scraper API / Browser API when structured results are not enough.