Collection from search engines and AI assistants fails sometimes. Sources refuse automated traffic, connections drop and pages arrive incomplete. A good integration expects that, distinguishes the kinds of failure and never records a failure as real data. This guide explains how ProtoScrape reports failures and how to handle them.

Two kinds of error

Request errors happen when you submit a job, before any collection. They return an HTTP error immediately and reserve no credits. Examples: an invalid engine (400), insufficient credits (402) or too many requests (429, with a Retry-After header).

Job failures happen during collection. The submission succeeded, so you have a job ID; the job then ends with status: "failed" or "expired" and an error object.

Job failure codes

Code Meaning What to do
upstream_blocked The source refused or challenged every attempt Retry later; avoid bursts of the same query
upstream_timeout The source did not answer in time Retry later
upstream_unavailable The source could not be reached Retry later; check the status page if widespread
no_usable_result A response arrived without a complete result Retry; if it persists, the query may have no answer
engine_failed Another or unknown failure Retry once; contact support if it repeats
deadline_exceeded The job did not finish within 30 minutes Submit a new job 
{
  "id": "job_…",
  "status": "failed",
  "charged_credits": 0,
  "error": {
    "code": "upstream_blocked",
    "message": "The source refused or challenged every collection attempt. Reserved credits were released."
  }
}

New codes may be added. Treat any code you do not recognize like engine_failed.

What is already retried for you

Each job gets up to three collection attempts within its 30-minute deadline before it fails. Within each attempt, collection may also try fresh sessions. You do not need to resubmit immediately after a single transient error; by the time a job reports failed, those retries are already spent.

Credits are released on every failure

Credits are reserved when a job is admitted and charged only on success. A failed, expired or cancelled job releases the full reservation and costs nothing. A consent page, challenge page or empty response is never billed as a result.

Retry safely with idempotency keys

Every submission carries an Idempotency-Key. Reuse the same key when a submission times out or your process restarts, and you get the original job back instead of a duplicate charge. Use a new key when you deliberately want a fresh collection after a failed job, because the original key will keep returning the failed job.

A simple policy:

  1. Submit with a key derived from your own record, for example study-42-query-7-2026-11-05.
  2. On a network error or 429, wait for Retry-After and resubmit with the same key.
  3. When the job ends failed with a retryable code, wait a few minutes and submit with a new key such as …-retry-1.
  4. Stop after a small number of attempts and record the observation as missing.

Keep failures in your dataset, honestly

Store every attempt with its status and error code. Exclude failed observations from rates and averages, and report coverage alongside results: “18 of 20 queries collected”. Replacing failures with zeros or “not found” creates false drops in trend charts.

Get notified instead of polling

Signed webhooks deliver job.succeeded and job.failed events, including the error code, as soon as a job finishes. For interactive use, the synchronous wait option returns quickly completing jobs in the submission response.

For background on why sources refuse automated traffic, read anti-bot protection explained. The API documentation lists every endpoint and error.

About ProtoScrape

We provide an independent API for collecting structured AI search observations. These guides explain our public product and workflows you can build with it.

Explore what ProtoScrape provides →