Teams collect Google search results to track rankings, study competitors, monitor brand visibility and compare classic results with AI answers. In 2026 there are three realistic ways to do it, and one of them is going away. This guide compares them on data quality, effort and cost so you can choose deliberately.
Option 1: Google's official search APIs
Google's Custom Search JSON API returns results from a Programmable Search Engine as JSON. It has been the default “official” answer for years, but it is closed to new customers and scheduled to be discontinued on January 1, 2027. Programmable Search Engines created since January 2026 can only search sites you specify, not the whole web.
Google points enterprise customers toward Vertex AI Search. That product is designed to search your own content and selected sites; it is not a drop-in replacement for observing the public Google results page that a user sees.
Even before the closure, the Custom Search results were not guaranteed to match google.com. For rank tracking, that difference matters: you want to measure what searchers actually see.
Choose this if: you already have access, need a small daily volume, and only search specific sites.
Option 2: build your own Google scraper
A homegrown scraper requests the results page and parses it. A prototype is quick. Production is not, for three reasons:
- Anti-bot protection. Google detects automated traffic and answers with rate limits (
429responses), consent pages and challenges. Our anti-bot explainer describes how these systems work. - Layout changes. Result markup changes without notice. A parser that silently returns zero results after a change is worse than one that fails loudly.
- Validation. Every response must be checked: was it a real results page, for the right market, with usable results? Without that check, blocked requests turn into fake “not ranking” data points.
You also carry the operational cost of the infrastructure and the responsibility for how your traffic behaves. Google's terms restrict automated access, so review them and your own risk tolerance before building.
Choose this if: you have engineering time to maintain it, modest volume and a tolerance for gaps.
Option 3: use a SERP API
A SERP API runs the collection for you and returns structured results. You trade a per-result price for not maintaining scrapers. When comparing providers, look past the headline price:
| Question | Why it matters |
|---|---|
| Are you charged for failures? | Paying for blocked or empty responses inflates real cost |
| Does a failure say why? | “Blocked” and “timed out” need different responses than “no results” |
| Can you get the raw page? | Lets you audit parsing and inspect features the JSON omits |
| How are retries handled? | Idempotent submission prevents paying twice for one query |
| Do unused credits expire? | Monthly expiry penalizes uneven workloads |
How ProtoScrape approaches it
ProtoScrape's google_serp engine returns US English desktop results as parsed JSON, with each organic result's position, title, URL, snippet and domain, plus People Also Ask questions and related searches. Add "output": "html" to receive the raw results page instead. See HTML vs JSON for when each is useful.
curl https://api.protoscrape.com/v1/jobs \
-H "Authorization: Bearer $PROTOSCRAPE_API_KEY" \
-H "Content-Type: application/json" \
-H "Idempotency-Key: rank-check-001" \
-d '{"engine":"google_serp","query":"best crm for small business"}'
A successful job costs one credit, from $1.50 per 1,000 results on the Scale plan to $2.90 on Developer. Failed jobs cost nothing and report a reason such as upstream_blocked or upstream_timeout. Unused credits roll over while your plan continues. The same API also collects Google AI Mode, ChatGPT and Perplexity answers, so organic and AI visibility can live in one dataset.
A quick decision guide
- Under a few hundred queries a day, with existing Custom Search access: keep it until 2027, then migrate.
- Large or long-running rank tracking: a SERP API is usually cheaper than the engineering time a scraper consumes.
- Research that compares Google with AI answers: prefer one source that collects both, so timestamps and markets line up.
Whatever you choose, store each observation with its query, timestamp, market and collection status, and keep failed observations as missing data rather than as rankings. The Google SERP data guide covers how to structure that dataset. ProtoScrape is independent and is not affiliated with Google.