Monitor scraping jobs

Job metrics

  • Scraped pages - Executed page count and total scheduled page count.
  • Scraped records - Rows produced by the sitemap.
  • Failed pages - Pages that returned a 4xx or 5xx response or did not load.
  • Empty pages - Pages that loaded successfully but produced no extracted data.
  • No-value pages - Pages where elements matched but no populated values reached the final dataset.

Inspect a job

The job inspection page includes lists of empty and failed pages. Screenshots and HTML are available for FullJS driver inspection cases. The Details tab includes the sitemap ID, scraping job ID, job configuration and scraping duration. Scraping duration excludes queue time and periods when the job status was not actively scraping.

Performance graph

While the job runs, the performance graph reports scheduled pages, executed pages, records, empty pages and failed pages in five-minute intervals. Graph data is retained for seven days.

Interpret progress

The scheduled-page count can continue increasing while navigation selectors discover more URLs. If execution stops progressing, check queue state, access failures, delays and selectors that repeatedly discover the same target.

Open problem pages directly from the jobs table

The page-count values in the scraping jobs table are direct links when a job contains affected pages.

Select the number under:

  • Failed pages to open the job's Failed pages view.
  • Empty pages to open the job's Empty pages view.
  • No value pages to open the job's No value pages view.

This is the quickest way to move from the jobs list to the affected URLs without first opening the general Inspect view.

Inspect empty and failed pages

Open a scraping job and select Inspect to investigate pages that did not produce the expected result.

The Empty pages and Failed pages tabs show information for the affected pages, including:

  • The page URL.
  • The failure reason, when available.
  • A screenshot and the HTML of the returned page, when Web Scraper captured one.

Use the URL, screenshot, and HTML together to determine what the scraper actually received. For example, a failed page may show an access-denied or challenge page instead of the expected website content. This helps distinguish website-access problems from selector or sitemap configuration problems.

When a screenshot is available, open Preview to inspect the returned page visually. The screenshot may make issues such as access-denied responses, CAPTCHA pages, consent screens or unexpected page states immediately visible.

Related