AI Sitemap Wizard

When to use the wizard

Use the wizard when the website displays repeated items in a listing, such as products, properties, jobs or directory records.

The wizard is designed for listing-page scraping. Use the Advanced Sitemap Builder when the sitemap requires custom category navigation, complex click actions or full manual control over the selector tree.

Create the listing-page sitemap

  1. Open the listing page where scraping should begin.
  2. Select the Web Scraper extension icon.
  3. Start the AI Sitemap Wizard.
  4. The Sitemap Wizard creates a sitemap using the page where it was opened as the start URL.
  5. The Sitemap Wizard names the sitemap using the website domain and creation date: example-com-YYYY-MM-DD. Additional sitemaps created for the same domain on the same date receive a numerical suffix, for example example-com-YYYY-MM-DD-2.
  6. The Sitemap Wizard automatically maps the repeated listing items on the page.
  7. Confirm the mapped listing items and continue to the listing navigation step.

The Sitemap Wizard creates the listing-page selectors from the automatically mapped items. New fields cannot be added manually on the listing page.

Add pagination or scrolling

The wizard attempts to detect whether the listing uses pagination, scrolling or both.

  1. Review the detected listing navigation.
  2. If pagination was not detected, manually select the button that opens the next listing page.
  3. If scrolling was not detected, manually enable or select the required scrolling behavior if needed.

Automatic detection does not work on every website. Confirm that the selected navigation matches how additional listing items are loaded.

Finish the sitemap or visit individual item pages

If all required data is available on the listing page, finish the sitemap after configuring pagination or scrolling.

If additional data is available only on individual item pages:

  1. Select the links that open the individual items.
  2. The Sitemap Wizard visits five example item pages and collects information from them.
  3. Review the selectors generated from the example pages.
  4. Rename or delete generated selectors as required.
  5. Add any additional fields from an individual item page using the point-and-click interface.

On individual item pages, the wizard supports two selector types:

  • Text for text-based values.
  • Image for image URLs.

Finish the sitemap when the required listing-page and individual-item fields have been added.

Related