Sitemap.xml Link selector

Supported sources

The selector supports standard XML sitemaps, compressed .xml.gz files and sitemap indexes that reference child sitemaps.

Configuration

  • Sitemap URLs - One or more XML sitemap locations.
  • Discover from robots.txt - Add sitemap URLs declared by the website.
  • URL regular expression - Include only target URL patterns.
  • Minimum priority - Optional filter for websites that use sitemap priorities consistently.

Selector structure

_root
└── sitemap_xml
    └── valid_page
        ├── title
        └── price

Fig. 1: Sitemap with Sitemap.xml link selector and wrapper element selector

Filter invalid page types

Use a URL pattern where possible. When URLs do not distinguish the page type, use an Element selector such as body:has(h1.product-title) as a wrapper beneath the Sitemap.xml Link selector.

Large sitemap sets

Split large XML sitemap sets into separate Web Scraper sitemaps when combined XML downloads exceed processing limits or create an impractical job size.

Related