Anyone who has run a scraper for more than a day has seen it: requests that worked yesterday now return errors, empty pages or a “verify you are human” screen. That is anti-bot protection doing its job. Understanding how it works helps you design collection that behaves well and, just as importantly, data that tells the truth when collection fails.
This article explains the defenses. It does not explain how to defeat them.
Why sites defend against bots
Automated traffic is not all bad; search engines crawl the web with bots. But sites also face credential stuffing, ticket scalping, fake account creation and traffic that simply overloads them. The OWASP Automated Threats to Web Applications project catalogs these patterns. Defenses are built to stop abuse, and ordinary scrapers get caught in the same net when they look or behave like abuse.
The main detection layers
Rate limits
The simplest defense counts requests from a client over time and refuses the excess. The standard response is HTTP 429 Too Many Requests (RFC 6585), often with a Retry-After header telling you when to try again. Respecting that header is both polite and practical: ignoring it usually escalates to harsher blocks.
Network reputation
Sites score where traffic comes from. Addresses belonging to data centers, or addresses that recently sent suspicious traffic, are treated with more suspicion than ordinary consumer connections. That is why the same request can succeed from one network and be challenged from another.
Client signals
A browser reveals a lot about itself: the headers it sends and their order, how it negotiates encrypted connections, the features its JavaScript engine supports, screen properties and more. Defenses compare these signals for consistency. A client claiming to be a mainstream browser while behaving like a script stands out.
Behavior
Humans scroll, pause and move irregularly. Automated clients often request pages at fixed intervals, skip the resources a real page loads, or visit thousands of pages in a pattern no person would. Behavioral models look for those differences over a session.
Challenges
When a system is unsure, it asks the client to prove itself: a CAPTCHA, a JavaScript computation, or an interstitial page that must be completed before content appears. A challenge is a request for verification, and you should not attempt to bypass verification you are not entitled to complete.
What a block looks like in your data
Blocks rarely announce themselves clearly. They show up as:
- Explicit refusals:
403or429status codes. - Substitute pages: a consent screen, challenge or “unusual traffic” page returned with a normal
200status. - Thin pages: a page shell with no results because content requires JavaScript or was withheld.
- Timeouts: connections that hang until your client gives up.
The second and third cases are the dangerous ones. A scraper that parses a challenge page and finds zero results may record “not ranking” or “no answer”, a false observation that looks like real data. Always validate that you received the page you asked for before extracting from it.
Collecting responsibly
- Ask for less. Collect only the pages and fields you need, at the lowest frequency that answers your question.
- Slow down when told. Honor
429andRetry-After, and back off further after repeated refusals. - Respect boundaries. Do not collect behind logins or paywalls, and read the site's terms and
robots.txt. - Record failures as failures. Keep a collection status on every observation, and exclude failed ones from metrics.
- Use an official API when one fits. It is the most reliable route when it provides the data you need.
How ProtoScrape reports blocks
Search engines and AI assistants actively manage automated traffic, so some collection attempts fail. ProtoScrape retries transient failures automatically. If a job still cannot be completed, it fails with a specific reason instead of a generic error:
| Error code | Meaning |
|---|---|
upstream_blocked |
The source refused or challenged every attempt |
upstream_timeout |
No answer before the deadline |
upstream_unavailable |
The source could not be reached |
no_usable_result |
A response arrived but was incomplete or empty |
Failed jobs cost nothing, and a challenge or consent page is never billed as a result. The errors and retries guide shows how to handle each code. ProtoScrape is independent and is not affiliated with the platforms it observes.