Receive AI search results with signed webhooks

Polling works well for a first API integration. When your application collects many prompts, a callback can save repeated status requests. ProtoScrape completion webhooks send your application an event when an admitted scraping job succeeds, fails, expires or is cancelled.

Successful events contain the complete structured result that ProtoScrape stored for the job. Failed, expired and cancelled events contain the job status and a null result. Delivery attempts and replays do not use scraping credits. A successful scraping job remains charged if your callback server is unavailable.

Connect your receiver

Open Webhooks in your workspace. Add a name and a public HTTPS URL. You can configure two endpoints per account. Endpoints receive all future terminal job events for that account, including admitted jobs from batches. Pending batch inputs that never become jobs do not produce job events.

Use a hostname on port 443, with no credentials, query parameters or fragment. Private addresses and redirects are not supported. Your endpoint will receive customer prompts and collected results; use a server that you control.

Save the signing secret when it appears. It is shown once. Store it in your server's secret manager and never send it to a browser or include it in a public repository. If you lose it, rotate the secret in the workspace and update your receiver.

Choose Verify & enable after installing your receiver. ProtoScrape sends a signed webhook.test event with a random challenge. After verifying the signature, respond with a successful HTTP status and this response header:

Protoscrape-Webhook-Confirmation: HMAC_SHA256_HEX(signing_secret, "verify." + challenge)

The secret is used as its literal UTF-8 string, including the whsec_ prefix. A matching response enables delivery. A normal 200 OK without the confirmation does not verify ownership. Verification events contain no customer job data and consume no credits.

Verify the raw request before processing

Each delivery includes Protoscrape-Event-ID, Protoscrape-Delivery-ID and Protoscrape-Signature headers. The signature has this form:

t=1793740000,v1=<64-character lowercase hexadecimal signature>

Compute HMAC-SHA256 over the timestamp, a literal period and the exact raw request body bytes. Compare signatures using a constant-time comparison. Reject timestamps outside a five-minute window around your server clock; keep that clock synchronized. Parse JSON only after validation. Re-serializing JSON changes the bytes and can invalidate the signature.

Here is a Node.js receiver skeleton using Express. persistOnce represents your durable database operation: save the event and any work to run in one transaction, with event.id as a unique key. Implement that operation before deployment.

import express from 'express';
import { createHmac, timingSafeEqual } from 'node:crypto';
import { persistOnce } from './your-durable-event-store.js';

const app = express();
const secret = process.env.PROTOSCRAPE_WEBHOOK_SECRET;
if (!secret) throw new Error('Webhook secret is required');
const mac = value => createHmac('sha256', secret).update(value).digest();

// Register this route BEFORE express.json().
app.post('/protoscrape/events', express.raw({ type: 'application/json', limit: '1mb' }), async (req, res) => {
  const match = /^t=(\d{1,12}),v1=([a-f0-9]{64})$/.exec(req.get('Protoscrape-Signature') || '');
  if (!Buffer.isBuffer(req.body) || !match || Math.abs(Date.now() / 1000 - Number(match[1])) > 300) {
    return res.sendStatus(401);
  }
  const signed = Buffer.concat([Buffer.from(match[1] + '.'), req.body]);
  if (!timingSafeEqual(mac(signed), Buffer.from(match[2], 'hex'))) return res.sendStatus(401);
  let event;
  try { event = JSON.parse(req.body.toString('utf8')); } catch { return res.sendStatus(400); }
  if (event.id !== req.get('Protoscrape-Event-ID')) return res.sendStatus(400);
  if (event.type === 'webhook.test') {
    if (typeof event.challenge !== 'string') return res.sendStatus(400);
    res.set('Protoscrape-Webhook-Confirmation', mac('verify.' + event.challenge).toString('hex'));
    return res.sendStatus(204);
  }
  try {
    await persistOnce(event.id, event); // Duplicates must succeed without repeating work.
    return res.sendStatus(204);
  } catch {
    return res.sendStatus(503); // Ask ProtoScrape to retry; do not acknowledge lost data.
  }
});
app.listen(process.env.PORT || 8080);

Acknowledge after durable acceptance and do slower processing in your own worker. ProtoScrape waits up to ten seconds for a response. Any 2xx status acknowledges a regular job event; redirects, other HTTP statuses, timeouts and connection errors are unsuccessful attempts. Response bodies are discarded.

Read an event

The event envelope contains id, type, api_version, created_at and data. The current event schema version is 2026-11-04. Job types are job.succeeded, job.failed, job.cancelled and job.expired.

For job events, data contains the same job fields as job history, plus a nested data field containing structured results or null. Successful Google SERP results include organic; successful AI results include answer_text and citations. Batch jobs retain batch_id and an optional tag. These are collected observations, and upstream answers can vary.

Use charged_credits to distinguish an actual success charge from a reservation. A terminal event always has zero reserved_credits. Failed jobs have zero charged_credits. Webhook delivery never resubmits a scraping job.

Handle retries and duplicates

Delivery is at least once, so duplicates are possible. A request can reach your server even if ProtoScrape does not receive its response. Events can also arrive out of order. Deduplicate by event ID, including after manual replay. Each retry receives a fresh timestamp and signature; its event ID and body stay the same. If both configured endpoints feed one shared database, deduplicate across them.

ProtoScrape makes up to six automatic attempts. After unsuccessful attempts, the backoff intervals are one minute, five minutes, thirty minutes, two hours and six hours. Queue capacity and maintenance recovery can delay those attempts further; the schedule is not a delivery-time SLA. A process interruption may consume an attempt even when delivery cannot be confirmed.

An endpoint automatically pauses after ten consecutive unsuccessful job-delivery attempts across its events. This protects the service when a receiver stays offline. Repair the receiver, enable or verify it again, and replay the events you need. Paused endpoints receive no new events; jobs completed while an endpoint is paused are still available through normal job history and exports during their retention window.

Inspect, replay and rotate

The workspace shows delivery state, attempt count, response status, next attempt time and event references. API keys can also read GET /v1/webhooks and the paginated GET /v1/webhooks/deliveries. Endpoint changes and replay require your signed-in account; they cannot be performed with a scraping API key.

Events and delivery history are retained for approximately seven days. The replay window ends exactly seven days after event creation, even if storage cleanup is still pending. You can replay a delivered, failed or skipped job event up to three times, to its original verified and enabled endpoint. Each replay starts a new bounded retry cycle, preserves the event body and ID, and adds no scraping charge.

Pausing or deleting an endpoint stops future sends; a request already in flight may still arrive. Rotating its secret pauses and un-verifies the endpoint. Update your receiver, then verify it again. Retained events sent afterwards are signed with the current secret. If a request from the old secret is already in flight, it may fail and need replay after the rotation.

Webhook delivery complements the job API and result exports. Keep job IDs and save successful results in your own system when you need them beyond the retention window.