Virtual column parser

Create a Virtual column

  1. Open the sitemap's Parser tab.
  2. Select + Add column at the bottom of the parser table.
  3. Enter a unique column name.
  4. Select one or more source columns.
  5. Configure a separator when several source columns are used.
  6. Save the Virtual column.
  7. Add parsers to the new Virtual column as required.

Configuration options

  • Name - The output column name. It must be unique within the parser configuration.
  • Source columns - One or more existing scraped columns used as the Virtual column input.
  • Separator - Text placed between source values when several source columns are selected. Escape sequences such as \n, \r and \t can be used.

Virtual columns use raw source values

A Virtual column receives the raw values from its source columns.

Parsers configured on the original source column do not modify the raw value supplied to the Virtual column. Parsers configured on the Virtual column also do not modify the source column.

This allows different transformations to be built from the same scraped source.

Split one scraped field into several columns

Virtual columns are useful when several data points are contained inside one scraped field.

Example source:

42 Example Street
Springfield, AB 12345

Create separate Virtual columns from the same source:

street       ← address_raw
city         ← address_raw
region       ← address_raw
postal_code  ← address_raw

Add a Regex match parser to each Virtual column so each one extracts only its required part.

This keeps the original address_raw value available while also producing structured fields.

Combine several source columns

Select more than one source column when a new field should combine existing values.

Example:

Source columns:
brand
product_name

Separator:
 - 

Result:
Example Brand - Widget 123

Choose a separator that is appropriate for the downstream dataset. The source fields remain unchanged.

Create a transformed copy of a field

Use a Virtual column when both the original and cleaned version should be retained.

price_raw        $24.99 incl. tax
price_clean      24.99

The Virtual column can copy price_raw and apply Regex match, Replace text or other parsers without changing price_raw.

Create a column with a default value

A Virtual column can be combined with Replace text to create a fixed output value based on an existing source field.

  1. Create a Virtual column.
  2. Select any appropriate existing source column.
  3. Add a Replace text parser.
  4. Enable Regex.
  5. Match the complete source value with [\s\S]+.
  6. Replace it with the required default value.

Example:

Text to replace: [\s\S]+
Text to place: active
Use Regex: Yes

Remove the raw source column from the final output

After the Virtual column has been validated, the original source column can be removed from the downloaded output with Remove column.

The Virtual column continues to receive the raw source data even when the source column is hidden from the final output.

Parser order and Virtual columns

The Virtual column itself cannot be swapped with another parser using drag and drop. Parsers attached to the Virtual column can still be ordered and execute sequentially within that Virtual column.

Build the Virtual column first, then order its child parsers according to the transformation required.

When to use a Virtual column instead of changing the sitemap

Use a Virtual column when the sitemap already extracts the correct source value and the change is purely about output structure.

Change the sitemap instead when:

  • The wrong page element is being extracted.
  • Separate fields can be captured cleanly with selectors at extraction time.
  • The navigation hierarchy is incorrect.

Keep extraction logic in the sitemap and post-processing logic in Parser where possible.

Source column limitation

A selector configured with Multiple Records in One Column cannot be used as a source/base column for a Virtual column. It will not be available for selection as a Virtual column source.

If the scraped values need to be used as the basis for a Virtual column, use a selector output configuration that creates a supported source column instead.

Troubleshooting

  • If a Virtual column is empty, inspect the raw source columns first.
  • If a source parser appears not to affect the Virtual column, remember that Virtual columns use raw source values.
  • If source columns are combined incorrectly, check their order and the configured separator.
  • If removing the source column causes confusion, confirm whether it was only removed from output or actually deleted from the sitemap.
  • Do not remove a Virtual column unless you are prepared to lose that Virtual column and its child parser configuration.

Related