Parser
When to use Parser
Use selectors to extract the correct source value. Use Parser when that value needs post-processing, such as removing text, extracting part of a string, cleaning whitespace, converting a timestamp, creating derived columns or removing unnecessary output columns.
Configure and preview parsers
- Open the sitemap's Parser tab.
- Find the column to process and select Add parser.
- Choose and configure the parser type.
- Preview the result and save the configuration.
The preview uses the first 10 scraped records by default and can be changed to preview up to 100 records. Values longer than 1,000 characters are abbreviated in the preview only; the underlying parsed/exported value is not truncated by the preview.
Parser execution order
Parsers execute from left to right in the order shown for each column. Each parser receives the output of the previous parser. Drag and drop parser buttons to change the order.
The same parser type can be added more than once when several simple transformations are easier to maintain than one complex rule.
Parser types
| Parser | Use |
|---|---|
| Append and prepend text | Add fixed text before or after a value. |
| Convert UNIX timestamp | Convert a UNIX timestamp to a formatted date and time. |
| Regex match | Extract matching parts of a value. |
| Replace text | Replace or remove text, optionally with Regex. |
| Remove whitespaces | Normalize spaces, tabs and line breaks. |
| Strip HTML | Remove HTML tags and optionally decode entities. |
| Virtual column | Create a new output column from one or more scraped columns. |
Virtual columns use raw source values
A Virtual column reads the raw values of its source columns. Parsers applied to a source column do not change the raw value used by the Virtual column, and parsers applied to the Virtual column do not change its source columns.
A normal source column can be removed from the final output while a Virtual column continues to use its raw value. Removing a Virtual column itself deletes that Virtual column and its child parsers.
Add the scrape time
Select + Add time scraped to add the scraping timestamp and format it using the required time zone and date/time format.
Recommended workflow
Scrape representative data first, then add one transformation at a time and verify the preview. Fix incorrect extraction in the sitemap; use Parser only when the raw value is correct and needs formatting or restructuring.