Advanced Sitemap Builder

Create the sitemap

  1. Open the page where scraping should begin.
  2. Open browser Developer Tools.
  3. Open the Web Scraper tab.
  4. Select Create sitemap.
  5. Enter a sitemap name and the start URL.

Build listing-page navigation

Build the navigation path before adding fields that depend on it. Category navigation can be added when the scraper must traverse categories or subcategories.

  1. Check whether the listing uses pagination.
  2. Create a Pagination selector when additional listing pages must be discovered.
  3. Navigate into the pagination selector so that the selectors created inside it execute recursively on each pagination page.
  4. Under Generate Selectors with AI, select Listing page to generate selectors for the repeated listing items, or create the selectors manually.
  5. Add any fields that were not generated.

Add detail-page extraction

The Listing page and Single item page generators can be combined at different levels of the selector tree.

  1. Create a Link selector and select the item links.
  2. Navigate into the Link selector in the selector tree.
  3. Under Generate Selectors with AI, select Single item page to generate selectors for the linked detail pages, or add the selectors manually.
  4. If a detail page contains another repeated list, such as customer reviews, select Listing page while navigated into the Link selector. The generated selectors are added beneath the detail-page branch.
  5. Continue nesting listing and link selectors when the website contains additional levels of repeated items and detail pages.
  6. Preview the generated selectors on several detail pages.

The Single item page option is available only when you are navigated into a Link selector. The Listing page generator can also be used within that linked-page context.

Understand the selector tree

A child selector runs in the page or element context returned by its parent. Place fields beneath the selector that provides the context they require.

_root
└── pagination
    └── product
        ├── name
        ├── price
        └── detail_link
            ├── sku
            ├── description
            └── review
                ├── reviewer
                ├── rating
                └── review_text

Related