Scrape multiple categories

Choose the category source

Categories can be supplied as multiple start URLs or discovered with Link selectors. Use multiple start URLs when the category pages are already known and use the same structure. Use Link Selector when categories should be discovered from a parent page.

Selector structure

_root
└── category_link
    └── pagination
        └── item
            ├── name
            └── detail_link

Fig. 1: Multiple link selectors for category navigation

Prevent duplicate traversal

  • Select category links only from the intended navigation area.
  • Avoid selecting parent and child category links when they lead to overlapping records.
  • Use URL filters or a more specific CSS selector where the navigation contains unrelated links.

Preserve category context

To include the category name in the output, wrap each category with an Element selector. Under it, add a Text selector for the category name alongside the Link selector that opens the category page.

Example:

_root
└── category_wrapper
    ├── category_name
    └── category_url
        └── record_wrapper

Related