Web scraping is the automated collection of information that a website shows to visitors. A program requests a page, reads the response and turns the parts you care about into structured data such as rows, JSON objects or database records. The same idea powers price monitoring, search-ranking research, academic studies, news archives and, more recently, tracking how AI assistants answer questions.
This guide explains how scraping works, when an official API is the better choice, and the limits a responsible collection project should respect.
How a scraper works
Most scraping projects follow the same four steps:
- Request. Fetch a page over HTTP, or load it in a real browser when the content only appears after JavaScript runs.
- Extract. Locate the fields you need in the HTML or in the data the page loads, for example a title, a price or a list of links.
- Validate. Check that the response is the page you expected and not an error page, a consent screen or a challenge. A scraper that stores whatever comes back produces silent gaps.
- Store. Save each observation with its context: the URL or query, the collection time, the market and the outcome.
The validation step is the one most often skipped, and it is where data quality is won or lost. A record that says “collection failed” is useful. A record that quietly treats a blocked page as “no results” corrupts every chart built on top of it.
Static pages, rendered pages and hidden data
Some pages deliver their full content in the first HTML response. Others send a minimal shell and fill it in with JavaScript, which means a simple HTTP request sees almost nothing. Many modern sites also load structured data from background requests that the page itself uses.
The right approach depends on the target. Rendering a full browser for every page is slower and more expensive, so it is worth checking first whether the information is already present in the initial response.
Scraping or an official API?
If a site offers an official API for the data you need, start there. APIs come with documented fields, stable formats, published limits and clear terms. Scraping makes sense when no API exists, when the API omits what users actually see, or when you need to observe a page exactly as it is displayed, which is often the point of search and AI-answer research.
Official APIs also change. Google's Custom Search JSON API is closed to new customers and scheduled to be discontinued on January 1, 2027, which is pushing many teams that tracked search results to look for alternatives. Our guide to collecting Google results compares the options.
Common uses
| Use | What is collected | Why observation matters |
|---|---|---|
| SEO rank tracking | Ordered search results for chosen queries | Rankings vary by market, device and time |
| AI visibility | Answers and cited sources from AI assistants | Answers change and are not exposed by a ranking API |
| Price and catalog monitoring | Product names, prices and availability | Displayed prices can differ from feeds |
| Research and journalism | Public posts, documents and listings | Archives preserve what was visible at a given time |
Where scraping stops
Being technically able to fetch a page does not settle whether you should. A responsible project considers four questions before collecting anything:
- Access. Collect only what is publicly visible without logging in. Do not bypass paywalls, authentication or access controls.
- Load. Keep request rates modest and back off when a site signals it is overloaded. A
429 Too Many Requestsresponse is an instruction to slow down, not an obstacle to defeat. - Rules. Read the site's terms and its
robots.txt, defined in RFC 9309. Robots rules are not access control, but they communicate the operator's expectations. - Personal data. Collecting information about people brings data-protection obligations in many jurisdictions. Minimize what you keep and how long you keep it.
The legal picture varies by country and by the facts of each case. In the United States, the Ninth Circuit's 2022 decision in hiQ Labs v. LinkedIn held that scraping publicly accessible profiles likely did not violate the Computer Fraud and Abuse Act's “without authorization” provision, while noting that other legal claims can still apply. This article is general information, not legal advice; take advice for your own use case.
Build or buy
Building a scraper for a stable, friendly site is a reasonable weekend project. Maintaining collection from large, frequently changing pages such as search engines and AI assistants is different: layouts change, sessions expire and rate limits tighten, and every failure has to be detected and reported honestly.
ProtoScrape handles that collection for Google Search, Google AI Mode, ChatGPT and Perplexity through one job API. You submit a query, follow a job ID and receive a structured result. Failed jobs cost nothing and report why they failed. ProtoScrape is independent and is not affiliated with the platforms it observes.