Troubleshoot Cloud scraping jobs

Identify the outcome

  • Failed page: the page could not be processed successfully because of an HTTP error, access problem or execution failure.
  • Empty page: the page loaded, but the sitemap produced no extracted record.
  • No-value page: matching elements were found, but no populated values reached the final dataset.
  • Incorrect data: the job ran, but the returned schema or values are wrong.

Compare environments

  1. Run the same sitemap and start URL locally.
  2. Confirm that Cloud has the intended sitemap version.
  3. Compare Fast driver and FullJS driver.
  4. Compare appropriate proxy locations when the website is geographic.
  5. Inspect FullJS driver screenshots for redirects, consent pages, login pages or access challenges.

Check timing and limits

Increase page load or interaction delay only when the required content appears after an asynchronous update. Also check URL-credit availability, parallel tasks, the 20,000 start URL limit, the 100-selector limit and the 10-minute continuous JavaScript execution limit per URL. This limit applies to JavaScript-driven actions such as clicking and scrolling, not to the total time spent extracting data from the URL.

Retry remaining pages

Cloud automatically re-scrapes empty and failed pages without charging additional URL credits. If pages remain after the job completes, correct any persistent access or sitemap problem before using Continue to schedule another attempt.

Use Inspect before changing the sitemap

When a job contains Empty or Failed pages, open Inspect before changing selectors or scraper settings. The relevant tabs show the affected URL, the failure reason when available, and a screenshot when one was captured.

The screenshot shows what Web Scraper actually received for that URL and can help identify access-denied pages, CAPTCHA challenges, consent screens or other unexpected responses.

Failure reasons

Failed pages can include Web Scraper-specific reasons such as Timeout and Website State Setup failures, as well as CAPTCHA detection or HTTP 4xx/5xx responses from the target website. A CAPTCHA does not always surface under the same failure reason.

See the Troubleshooting reference for explanations and recommended checks for each failure reason.

Related