Webhook notifications

When to use webhooks

Use a webhook when another system needs to act after a scraping job finishes, stops or fails. Common examples include downloading the completed dataset through the API, importing records into a database, starting a transformation process, or triggering another internal workflow.

Without a webhook, an automated integration typically has to request the job status repeatedly until it changes. With a webhook, your system can wait for Web Scraper Cloud to notify it and make follow-up API requests only when they are needed.

Webhook-based API workflow

  1. Start the scraping job through the API, Scheduler, or Web Scraper Cloud UI.
  2. When the job finishes, Web Scraper Cloud sends a webhook to your endpoint.
  3. Your endpoint accepts the event and returns an HTTP 2xx response within 10 seconds.
  4. Use scrapingjob_id with the Cloud API to download and process the scraped data.

The webhook is the completion trigger. The API remains the interface used to retrieve the job data and perform follow-up actions.

Configure a webhook endpoint

  1. Create a public HTTPS endpoint that can receive POST requests.
  2. Configure the endpoint from the Web Scraper Cloud API page.
  3. Test the endpoint and confirm that it returns a successful response.

When a webhook is sent

Web Scraper Cloud sends a webhook when a scraping job reaches one of these final statuses:

  • finished
  • stopped
  • failed

If empty or failed URLs are rescheduled with Continue, a fresh webhook can be sent for the same scraping job after the additional processing reaches a final status.

Webhook payload

The webhook is sent as a form-encoded HTTP POST containing scraping-job metadata.

scrapingjob_id=1234
status=finished
sitemap_id=12
sitemap_name=product-catalog
custom_id=batch_8472
Field Use
scrapingjob_id The scraping-job identifier used for follow-up API requests and data downloads.
status The final job status: finished, stopped or failed.
sitemap_id The Cloud sitemap identifier.
sitemap_name The sitemap name.
custom_id A custom identifier supplied with the scraping job, when used.

The webhook does not contain the scraped dataset. Use scrapingjob_id with the Web Scraper Cloud API to retrieve the data.

Use custom_id to correlate jobs

If your application starts a job with custom_id=batch_8472, the webhook returns the same custom_id. Your application can then associate the finished scraping job with the correct internal batch, request or workflow.

Response requirements and retries

Your endpoint must return an HTTP 2xx response within 10 seconds. Do not make the webhook request wait while a large dataset is downloaded or imported.

Web Scraper Cloud retries the webhook when the endpoint does not respond within 10 seconds or returns an HTTP status code of 300 or higher. The first retry is sent after 5 seconds and the second after 10 seconds.

Because retries can deliver the same notification again, and Continue can create another valid notification for the same scraping job, webhook processing should be idempotent. Track the job and status before starting downstream work so the same event does not create duplicate imports.

Handle the data import

For simple integrations, the webhook handler can acknowledge the notification and continue processing immediately. For more resilient workflows, queue the scrapingjob_id and let a worker download and import the data separately.

The queue approach is preferable when imports are long-running or need independent retry handling.

PHP on the fly

PHP queue handler

This example returns a successful response immediately and then processes the scraping-job data in the same PHP request.

<?php
use WebScraper\ApiClient\Client;
use WebScraper\ApiClient\Reader\JsonReader;

// Validate that the request came from Web Scraper.
// One option is to send a secret token in the webhook URL.

$scrapingJobId = (int) $_POST['scrapingjob_id'];
$status = $_POST['status'];
$sitemapId = (int) $_POST['sitemap_id'];
$sitemapName = $_POST['sitemap_name'];
$customId = $_POST['custom_id'];

// Return a successful response and continue processing.
ignore_user_abort(true);
header('Connection: close');
header('Content-Length: '.ob_get_length());
ob_end_flush();
ob_flush();
flush();

// For production use, consider moving the import to a queued job.
$client = new Client([
    'token' => 'YOUR API TOKEN',
]);

$outputFile = "/tmp/scrapingjob-data{$scrapingJobId}.json";

try {
    $client->downloadScrapingJobJSON($scrapingJobId, $outputFile);

    $reader = new JsonReader($outputFile);
    $rows = $reader->fetchRows();

    foreach ($rows as $row) {
        // Import records into your database.
        // Bulk imports are recommended for larger datasets.
    }
} finally {
    unlink($outputFile);
}

$client->deleteScrapingJob($scrapingJobId);

This example represents the worker that processes a scrapingjob_id previously placed into a queue by the webhook handler.

<?php
require "../vendor/autoload.php";

use WebScraper\ApiClient\Client;
use WebScraper\ApiClient\Reader\JsonReader;

$apiToken = "API token here";
$scrapingJobId = 500; // Read the scraping job ID from the queued job.

$client = new Client([
    'token' => $apiToken,
]);

$outputFile = "/tmp/scrapingjob-data{$scrapingJobId}.json";

try {
    $client->downloadScrapingJobJSON($scrapingJobId, $outputFile);

    $reader = new JsonReader($outputFile);
    $rows = $reader->fetchRows();

    foreach ($rows as $row) {
        // Import records into your database.
        // Bulk imports are recommended for larger datasets.
    }
} finally {
    unlink($outputFile);
}

$client->deleteScrapingJob($scrapingJobId);

Security

  • Use HTTPS for the webhook endpoint.
  • Validate required fields such as scrapingjob_id and status before processing the event.

Troubleshooting

The webhook is delivered more than once

Confirm that the endpoint returns 2xx within 10 seconds. Retries can duplicate a delivery, and Continue can generate another valid notification for the same scraping job. Keep processing idempotent.

Web Scraper keeps retrying the webhook

Move long-running downloads and imports out of the webhook request. Return 2xx first, then process the data immediately after the response or through a queue.

The webhook arrives but no data is imported

Check the follow-up API or queue worker separately. A successful webhook response confirms only that the notification was received.

Related