Scheduler
Configure a schedule
- Open the sitemap in Web Scraper Cloud.
- Open the Schedule tab.
- Select the scheduler time zone and schedule type.
- Configure the timing.
- Configure Driver, Request interval, Page load delay, Proxy and Priority.
- Save the schedule.
Request interval and Page load delay have a minimum value of 500 ms. The default values are a good starting point; lower values can be tested when faster execution is required.
Schedule types
| Type | Use |
|---|---|
| Daily | Select one or more weekdays and a specific time. |
| Interval | Run repeatedly using a minutes or hours interval. Hour intervals align to round clock hours. |
| Custom Cron expression | Define a recurring schedule that cannot be represented by Daily or Interval. |
All schedule timing is evaluated using the selected scheduler time zone.
Previous scheduled job still running
If the next scheduled time arrives while the previous scheduled job for that sitemap is still running, the next run waits until the previous one finishes. It then starts when the normal parallel-task rules allow it.
A schedule interval shorter than the sitemap's normal execution time therefore does not increase the actual run frequency.
Sitemap version used by scheduled jobs
The next scheduled job is created from the latest saved or synchronized sitemap. Once a job has already been created and placed in the queue, later sitemap changes do not modify that queued job. It runs with the sitemap version captured when the job was created.
Priority and parallel tasks
Scheduled jobs use the configured Priority and the account's normal parallel-task capacity. Priority changes the order of waiting jobs; it does not interrupt jobs that are already running.
Before scheduling
Validate the same driver, proxy, Request interval and Page load delay with Test scrape or a normal job. Also choose an interval that fits the sitemap's typical duration and expected page usage.