Troubleshooting

Choose the observed problem

Diagnostic order

  1. Open the exact source URL manually.
  2. Confirm the expected data exists on that page.
  3. Use Element Preview.
  4. Use Data Preview.
  5. Run one or a few URLs locally.
  6. Compare the Cloud sitemap version and driver.
  7. Inspect failed, empty and no-value pages.
  8. Check for source-site changes.

Failure reasons and HTTP status codes

Pages shown under Failed pages can fail for different reasons. Use the failure reason together with the affected URL and screenshot, when available, to determine what Web Scraper actually received.

Failure reasons fall into two broad groups:

  • Web Scraper execution reasons, such as CAPTCHA detection, JavaScript Timeout, or a Website State Setup action that cannot find its required element.
  • HTTP errors returned by the target website, typically a 4xx or 5xx response.

Web Scraper failure reasons

Failure reason What it means What to check
CAPTCHA or another access error A CAPTCHA may be detected and reported as CAPTCHA, but depending on the response it can also surface as an HTTP or other access-related failure. Inspect the screenshot and returned response before diagnosing the failure from the reason alone. Review proxy type, location and target-site access behavior.
Timeout The page exceeded the 10-minute continuous JavaScript execution limit. Inspect long-running scrolling or Element Click actions. Remove unnecessary interactions or reduce the number of repeated JavaScript actions.
Couldn't find input - input#customer-email:visible A Website State Setup action could not find an element required by the configured sequence. Compare the screenshot with the Website State Setup configuration. Confirm that the required input, button, link or other element exists in the returned page state and that its selector is still valid.

The Website State Setup message varies with the configured action. For example, the missing element may be an input, button, link or another selector used by the setup.

HTTP 4xx responses

A 4xx response means the target website rejected or could not fulfill the request. Common examples include:

Status Typical meaning What to check
400 Bad Request The website rejected the request as invalid. Check the URL and whether the site requires a particular page state, cookie or request flow.
401 Unauthorized The page requires authentication or the current authenticated state is not accepted. Check Website State Setup, login state and whether the page can be accessed with the configured workflow.
403 Forbidden The website refused access to the request. Inspect the returned page. A 403 can represent an access restriction, bot block, geo block or another site-specific denial.
404 Not Found The requested URL does not exist at that location. Confirm that the URL is current and that the sitemap is not following stale or malformed links.
408 Request Timeout The target server or an intermediary timed out while handling the request. Retry the page and check whether the issue is temporary. This is different from Web Scraper's Timeout reason for the 10-minute continuous JavaScript limit.
429 Too Many Requests The target website is rate-limiting requests. Review request interval, proxy configuration and the target site's access behavior.

HTTP 5xx responses

A 5xx response means the target website or an upstream service failed while processing the request. Common examples include:

Status Typical meaning What to check
500 Internal Server Error The target website encountered an internal error. Retry and compare the result with the website in a normal browser. A persistent 500 may be specific to the requested page or access path.
502 Bad Gateway An upstream server returned an invalid response. Often temporary. Retry and inspect whether the target site itself is unstable.
503 Service Unavailable The target service is temporarily unavailable or refusing traffic. Retry later and inspect whether the site is under maintenance, overloaded or restricting the request.
504 Gateway Timeout An upstream service did not respond in time. Retry the page and check whether the issue persists across requests or proxy locations.

HTTP status codes describe the response returned by the target website or its infrastructure. The exact page content can still vary, so use the screenshot and returned URL alongside the status code rather than diagnosing the issue from the number alone.

Related