Scraping product reviews: build a feedback dataset

Scraping product reviews gives you useful customer feedback when each review retains its product identity, source, publication date and collection context. Text and star ratings alone cannot tell you whether a complaint is new, duplicated across retailers or about a different variant.

Define the records and coverage before building the scraper, so recurring exports can preserve changes without counting the same feedback repeatedly.

Define the question and collection scope

Suppose you want to identify bottle models with recurring complaints about leaking lids in reviews published during the previous three months. You need complete review text, dates, product matches and a consistent collection scope.

A product's overall rating cannot supply that evidence. Schema.org distinguishes aggregate ratings from individual reviews, with separate ratingCount and reviewCount properties. A displayed total of thousands of ratings does not establish that thousands of written reviews are available. Store displayed averages and counts as dated product observations, separate from individual review rows.

Specify the product list, sources, storefronts, languages, publication window and review types. Decide whether rating-only entries belong in the project. They can contribute to a rating distribution without supplying text for complaint analysis.

Prefer an authorised export, API or licensed feed when it provides the required records and reuse rights. For website collection, review the access conditions, terms and privacy requirements before implementation. Avoid collecting reviewer names or profile details when the task only requires product feedback.

Design a product review schema before scraping

Separate review identity from repeated observations. Review R123 appearing again tomorrow is existing feedback observed again, possibly with changed text, rating or helpful votes.

A flat export is enough for a pilot. For recurring collection, keep a current review record, dated observations and page associations as linked records. Observations hold mutable fields and collection times; page associations preserve where a review appeared. Overwriting the URL whenever it appears on another product page would lose that history.

Choose fields that the source exposes and your analysis needs. Badges, helpful votes and modification dates are optional.

Fields	What to preserve
source_namespace, source_review_id	Identity within a documented website or provider-account scope
page_product_id, page_variant_id	The product page and selected variant where the review appeared
reviewed_product_id, reviewed_variant_id, review_scope	The explicitly reviewed subject, including family-level or unknown scope
canonical_product_id, match_status	Your cross-source product match and its confidence or review status
title_raw, review_text_raw	Original wording, separate from cleaned or translated text
rating_raw, rating_value, rating_min, rating_max	The published rating and original scale
published_at_raw, published_at, date_precision	Source publication date without invented precision
source_url, review_url, origin_source	Observation location, direct review link and disclosed origin
observed_at, run_id, collection_scope_id	Collection time, run and associated scope settings
verified_purchase_raw, incentive_raw, helpful_votes	Published disclosures and mutable counts when available
text_status, source_modified_at	Text completeness status and any published modification time

The following mapping uses your chosen schema names, not a set of automatically supplied export columns. All worked values in this mapping are synthetic.

Field	How to capture it in Web Scraper	Where
source_review_id	Use an Element Attribute selector on the review card's ID attribute, such as data-id.	Web Scraper
source_namespace	Create a Virtual column from a non-empty review ID, then use Replace text with Regex ^[\s\S]+$ to set the fixed value shop_uk. To produce the combined key shop_uk:R123, prepend shop_uk: to a separate Virtual copy of the raw ID with Append and prepend text.	Web Scraper
page_product_id, page_variant_id	Extract the displayed page identity with Text selectors or Element Attribute selectors where those values are explicitly exposed.	Web Scraper
reviewed_product_id, reviewed_variant_id	Extract the subject explicitly identified inside the review, using Text or Element Attribute selectors. Leave absent attribution empty.	Web Scraper
review_scope	Assign product, variant, family or unknown scope from the captured attribution and documented pooling behaviour.	Downstream
canonical_product_id, match_status	Join the extracted product identity to your product mapping table and retain its match status.	Downstream
title_raw, review_text_raw	Use Text selectors inside the review wrapper, with one value per field.	Web Scraper
rating_raw	Use a Text selector, or an Element Attribute selector for aria-label or data-rating.	Web Scraper
rating_value	Create a Virtual copy of rating_raw and apply Regex match. For the synthetic value 4 out of 5 stars, ^[0-9]+ returns 4.	Web Scraper
rating_min, rating_max	Attach constants for the source's confirmed rating scale during import.	Downstream
published_at_raw	Use a Text selector for the published date as displayed.	Web Scraper
published_at, date_precision	Parse textual dates and retain their actual precision. Parser's Convert UNIX timestamp handles UNIX timestamps, not a general written date such as 12 August 2026.	Downstream
review_url	Use an Element Attribute selector on the review link's href, when present.	Web Scraper
source_url	Retain the page URL captured with the record and map it to this field during import. Preserve the observed page, including relevant variant context, rather than deriving a URL from the review ID.	Downstream
origin_source	Use a Text or Element Attribute selector for an explicitly displayed syndication attribution or origin link.	Web Scraper
observed_at	Select + Add time scraped, then choose the time zone and date/time format.	Web Scraper
run_id, collection_scope_id	Attach the export's job identity and your collection-settings identity during import.	Downstream
verified_purchase_raw, incentive_raw	Use Text selectors for published badges or statements; leave them empty when absent.	Web Scraper
helpful_votes	Extract the displayed count with a Text or Element Attribute selector, then normalise its number if needed.	Web Scraper
text_status	Classify the result after checking full-text availability and extraction outcomes.	Downstream
source_modified_at	Capture an explicitly published modification value with Text or Element Attribute selectors; normalise textual dates downstream.	Web Scraper

Keep the raw columns unchanged and apply derived-value parsers to separate Virtual columns so the original text remains available. Parsers execute left to right within a column, while a Virtual column reads its source columns' raw values rather than their parsed outputs.

Preview the transformations on the first 10 records, increasing the preview to as many as 100 when necessary, then inspect a downloaded sample containing less common values. Where a source does not expose an actual page URL or identifier, retain that limitation in the downstream record instead of manufacturing a value.

Treat an absent verified-purchase badge as unknown unless the source defines it as false. A date without a time should remain a date. Adding midnight UTC invents precision the source did not provide.

Use a text status such as complete, truncated, rating_only, extraction_failed or unknown. Set rating_only only when the source establishes that no text was submitted. A blank extracted body could also mean that a selector failed or content did not load.

Preserve raw text before cleaning, translation or classification, with separate fields and versions for those transformations. Apply appropriate access and retention controls to raw copies, and minimise unnecessary personal information in analytical copies.

Verify which product the review concerns

A selected variant does not establish the subject of every review in its widget. Bazaarvoice, for example, documents product families that share reviews among related products, with each review's ProductId identifying its original subject.

In a synthetic example, review shop_uk:R123 appears on a blue 750 ml bottle page but explicitly identifies the black 500 ml bottle. Store the black 500 ml bottle as the reviewed variant and the blue 750 ml page as an observed association. Seeing it tomorrow adds an observation. Finding it on another product page adds another association.

Preserve explicit review-level attribution even when the widget pools feedback. Where the source only establishes family-level scope, retain that scope and leave the variant unknown. Do not infer it from the page selection.

Across retailers, maintain a product mapping table using published GTINs, manufacturer part numbers or model identifiers, supported by brand and variant attributes. Retailer SKUs need their own namespaces. Keep uncertain matches out of variant-specific comparisons until resolved.

Delivery complaints also need their own attribution. A review does not automatically describe the currently displayed seller or establish a fault in the bottle.

The synthetic sample below combines two job exports after downstream mapping, including assignment of text_status. blue_750 and black_500 are synthetic variant IDs; unknown means that the review does not establish a variant, and ∅ represents an empty field rather than literal exported text. Review text is shortened for display, without implying that the collected bodies were truncated.

source_namespace	source_review_id	page_variant_id	reviewed_variant_id	rating_raw	review_text_raw (shortened)	published_at_raw	observed_at	text_status	helpful_votes	origin_source
shop_uk	R123	blue_750	black_500	2 out of 5 stars	Lid leaks in my bag.	2026-08-12	2026-08-13T09:00:00Z	complete	2	∅
shop_uk	R123	blue_750	black_500	2 out of 5 stars	Lid leaks in my bag.	2026-08-12	2026-08-14T09:00:00Z	complete	5	∅
shop_uk	R123	black_500	black_500	2 out of 5 stars	Lid leaks in my bag.	2026-08-12	2026-08-14T09:00:00Z	complete	5	∅
shop_eu	E908	black_500	black_500	2 out of 5 stars	Lid leaks in my bag.	2026-08-12	2026-08-14T09:00:00Z	complete	∅	shop_uk:R123
shop_uk	R124	blue_750	unknown	5 out of 5 stars	∅	2026-08-12	2026-08-14T09:00:00Z	rating_only	0	∅
shop_uk	R125	blue_750	unknown	1 out of 5 stars	∅	2026-08-13	2026-08-14T09:00:00Z	extraction_failed	∅	∅
shop_uk	R126	black_500	black_500	4 out of 5 stars	Easy to clean.	12 August 2026	2026-08-14T09:00:00Z	complete	0	∅

1. Row 1 creates the source review shop_uk:R123 and its first observation on the blue page.
2. Row 2 adds a new observation with five helpful votes, without creating new feedback.
3. Row 3 adds the black-page association for the existing review, whose reviewed variant was already black_500.
4. Row 4 is a cross-source cluster candidate supported by the synthetic origin attribution; retain its separate retailer occurrence while checking the relationship.
5. Row 5 is a source-confirmed rating-only entry, retained for suitable rating analysis and excluded from text analysis.
6. Row 6 goes back for collection because missing extracted text is a known failure, not a confirmed rating-only submission.
7. Row 7 keeps 12 August 2026 as the publication date 2026-08-12, without adding a publication time.

Extract complete records through the required page states

Create and test the sitemap in the browser extension before Cloud automation. Use an Element selector with Multiple enabled for repeated review wrappers, then put review-level rating, text and date selectors beneath it.

Independent lists of ratings and text can silently misalign when joined by position if one card lacks a title or body. The wrapper keeps those fields together. Check that the rating selector returns the individual score rather than the product aggregate above the list.

Test a small set of pages covering no reviews, many reviews, missing titles, long text and variant information. Inspect the interactions needed:

• Check whether review-tab content is already extractable before adding a click.
• Compare exports with expanded reviews. Add the required interaction when expansion supplies otherwise unavailable text.
• Use a Pagination selector for Next or Load more controls, or Element scrolling for reviews loaded by scrolling. The pagination and scrolling guide explains those configurations.

This synthetic selector tree shows the product-summary and review branches. product_summary is an Element wrapper for the displayed product-level fields, with Multiple disabled; store that branch as dated product observations downstream, separately from the review records. The tree does not imply that one export creates separate database tables.

_root
├── product_summary (Element; Multiple disabled)
│   ├── displayed_average (Text)
│   └── visible_review_total (Text)
└── pagination (Pagination)
    └── review (Element; Multiple enabled)
        ├── review_id (Element Attribute: data-id)
        ├── rating (Element Attribute: data-rating)
        ├── title (Text)
        ├── text (Text)
        ├── date (Text)
        ├── reviewed_variant (Text)
        └── verified_badge (Text)

Try Auto first; use Link for a numbered Next control with an href, or Click multiple times on next/more button for a Load more control that appends reviews.

Use the Element selector's Scroll option when scrolling loads reviews without pagination controls; scrolling stops when it finds no new matching elements or reaches Element limit, while the requested period determines the dates retained and whether that limit reaches far enough back.

Scrolling or click-based Pagination requires FullJS, because Cloud does not allow those workflows on Fast; the collection period is not an automatic date-based Scroll stop setting.

Verify later pages and full text, not just the first visible cards. If the source stops supplying reviews at an access boundary or cap, record the resulting sample limitation.

In Cloud, use the FullJS browser driver when the workflow requires JavaScript or interactions. Fast extracts returned HTML without running page JavaScript and cannot run scrolling or Element Click workflows.

Record coverage and stopping reasons

Keep a collection manifest for each product and run: requested period, locale, sort, filters, visible total and its meaning, pages or batches visited, unique reviews collected, oldest and newest publication dates, and stopping reason.

Use reasons such as period_reached, no_more_results, configured_limit, access_restricted and execution_error. A product with zero reviews also needs an explicit outcome. Otherwise, no feedback and a missed product look identical in the output.

A Web Scraper job does not write these stopping-reason labels for you. Derive the manifest downstream from its outputs and the saved collection settings:

• Compare collected unique reviews with the captured visible total, after checking that both describe the same population.
• Retain the oldest and newest published_at_raw values; compare parsed dates downstream rather than sorting their raw strings alphabetically.
• Inspect the job's Failed and Empty page counts alongside its returned records.
• Check the number of collected review elements against the configured Element limit, before deduplication hides repeated appearances.

Signal in the job output	Likely stopping reason	Check next
The oldest collected review reaches or crosses the requested start date, and navigation is consistent with newest-first order	period_reached	Verify the ordering, check for pinned reviews and retain only the declared date window downstream.
The unique count matches the relevant visible total, dates are plausible and no access or loading gap is apparent	no_more_results	Check that the total covers the same language, review types and product scope; an exhausted accessible list can still be capped by the source.
The affected page returns the configured maximum number of review elements	configured_limit	Compare the oldest date with the required start date; raise the limit for that product if it stopped too early.
Failed pages include CAPTCHA, access-denied, 403 or 429 responses	access_restricted	Identify the affected products and access conditions; record the missing coverage before choosing an appropriate collection route.
Failed or incomplete pages show timeouts, 5xx responses or other loading failures, or expected records disappear	execution_error	Re-run the affected product and compare its returned fields and counts with a working page; counts alone do not identify the cause.
Click multiple times on next/more button stops after no new records are scraped	no_more_results or an unresolved loading/extraction failure	Compare with the relevant visible total and oldest date, then re-run that one product with a longer Page load delay. Additional records indicate premature stopping; an unchanged count still needs comparison with the expected source coverage.

Treat these as evidence-based classifications, with unknown retained where the signals do not establish a cause. A global job count can conceal one incomplete product, so keep the per-product checks alongside the job totals.

Check why loading stopped. No new records could indicate exhaustion, a delayed response or a broken selector. Before using dates as a stopping rule, verify that later batches move backwards through time and account for pinned or reordered reviews.

Compare counts only within matching populations. A total covering all languages, variants or rating-only entries may not match a filtered written-review collection. Document the difference before calculating completeness.

Newest-first and most-helpful collections are different samples. Keep their rules visible in reports. Schoenmueller, Netzer and Stahl found that consumers with extreme evaluations have a greater tendency to post reviews. Even a complete collection of accessible reviews does not represent every buyer.

Deduplicate while retaining source evidence

Use (source_namespace, source_review_id) as the preferred source key after confirming the ID's scope. Repeated observations update that review's history rather than create new feedback.

If stable IDs are unavailable, create conservative match candidates using product identity, source date, rating and original text. Keep matching evidence and allow corrections. Edits can change a text fingerprint, while unrelated buyers can both write “Leaks”.

Handle cross-source duplication separately. Bazaarvoice documents syndication that shares reviews from brand sites to retailers. Several retailer appearances can therefore represent one underlying review.

Use disclosed origin information, original IDs or links to cluster confirmed copies, while preserving each source occurrence. Similar text alone is insufficient to establish common identity or fraud, especially for short comments and translations.

Choose the counting unit for each report. Distribution analysis may count appearances; complaint analysis usually needs distinct review clusters. Report unresolved matching uncertainty, and do not describe those clusters as a verified count of unique customers.

Preserve edits as versions or observations. Refresh an overlapping publication window and periodically revisit older material when edits matter. A missing review in one run is not proof that the platform deleted it. Perform merging, version handling and deduplication downstream from the scraping exports.

The following synthetic PostgreSQL example imports a mapped projection of one job's CSV. A downstream import step adds the job ID, normalises the rating and assigns text_status; retain the remaining raw export fields alongside this projection.

The example assumes existing downstream tables with these constraints:

• review_observations has the nine columns listed in COPY, in that order, and a unique key on (run_id, source_namespace, source_review_id, source_url, page_variant_id).
• current_reviews has the six columns in its insert and a primary key on (source_namespace, source_review_id).
• review_page_associations has a unique key on its four inserted columns.

Use non-null identity, page URL, page-variant and observation-time fields; a defined unknown page-variant value can preserve an association whose variant is unavailable. The synthetic CSV path in COPY is a database-server path.

BEGIN;
CREATE TEMP TABLE incoming_reviews (LIKE review_observations) ON COMMIT DROP;
COPY incoming_reviews (run_id, source_namespace, source_review_id, source_url, page_variant_id,
                      observed_at, rating_value, review_text_raw, text_status)
FROM '/data/reviews/job-2026-08-14.csv' WITH (FORMAT csv, HEADER true);
INSERT INTO review_observations SELECT * FROM incoming_reviews WHERE true
ON CONFLICT (run_id, source_namespace, source_review_id, source_url, page_variant_id) DO NOTHING;
WITH ranked AS (
  SELECT *, MIN(observed_at) OVER w AS first_in_job,
    ROW_NUMBER() OVER (PARTITION BY source_namespace, source_review_id
      ORDER BY observed_at DESC, source_url, page_variant_id) AS rn
  FROM incoming_reviews WHERE text_status IN ('complete', 'rating_only')
  WINDOW w AS (PARTITION BY source_namespace, source_review_id)
)
INSERT INTO current_reviews (source_namespace, source_review_id, first_seen_at, last_seen_at, rating_value, review_text_raw)
SELECT source_namespace, source_review_id, first_in_job, observed_at, rating_value, review_text_raw FROM ranked WHERE rn = 1
ON CONFLICT (source_namespace, source_review_id) DO UPDATE SET
  first_seen_at = LEAST(current_reviews.first_seen_at, EXCLUDED.first_seen_at),
  last_seen_at = GREATEST(current_reviews.last_seen_at, EXCLUDED.last_seen_at),
  rating_value = CASE WHEN EXCLUDED.last_seen_at >= current_reviews.last_seen_at THEN EXCLUDED.rating_value ELSE current_reviews.rating_value END,
  review_text_raw = CASE WHEN EXCLUDED.last_seen_at >= current_reviews.last_seen_at THEN EXCLUDED.review_text_raw ELSE current_reviews.review_text_raw END;
INSERT INTO review_page_associations (source_namespace, source_review_id, source_url, page_variant_id)
SELECT DISTINCT source_namespace, source_review_id, source_url, page_variant_id FROM incoming_reviews WHERE true
ON CONFLICT (source_namespace, source_review_id, source_url, page_variant_id) DO NOTHING;
COMMIT;

Run this downstream after each automatic export, because each export contains one job's dataset. The ranked selection prevents a single upsert from attempting to update the same review twice when several pages contain it; the timestamp comparison prevents an older job from replacing newer content.

Failed, truncated and unknown text observations remain in history but do not replace accepted current content, so the current table's first/last-seen times describe accepted observations. Cross-retailer review clustering remains a separate step.

Separate feedback labels from source facts

A bottle review can praise insulation, criticise the lid and mention late delivery in the same paragraph. Its overall star rating cannot separate those experiences.

Create aspect labels such as lid_leak, insulation, cleaning and delivery, allowing several labels per review. Distinguish a complaint from a negated statement such as “does not leak”. Store each label with its supporting text span and classification version.

Use a written rubric before applying lid_leak. Every review text in this rubric is synthetic.

Review text (synthetic)	Label	Why
“The lid leaks even when it is closed properly.”	lid_leak	Explicit leakage complaint attributed to the lid under ordinary use.
“It does not leak at all.”	No lid_leak	The sentence denies leakage.
“It leaks if you overfill it.”	lid_leak, conditional	Retain the leakage mention and its overfilling condition; do not present it as an ordinary-use defect.
“The courier crushed the parcel and cracked the bottle, so it leaked.”	delivery	The stated cause is delivery damage, with no evidence of a lid fault.
“Wet bag again.”	uncertain	The text does not establish the source of the moisture or a lid problem.

Audit a varied manual sample across products, ratings and languages before using automated labels. Include mixed sentiment, negation and service complaints, with an uncertain outcome for ambiguous cases.

In a synthetic audit of 60 reviews, an automated rule marks 12 as lid_leak. Manual checking confirms 10 of those labels and finds three additional leak mentions that the rule missed, giving 10 true positives, two false positives and three false negatives:

• Precision: 10 / 12 × 100 = 83.3%.
• Recall: 10 / (10 + 3) × 100 = 76.9%.

Set an acceptance rule before deployment; for this synthetic pilot, require at least 90.0% precision and 90.0% recall, so the observed disagreement sends the rule back for revision before it is applied to the 120-review dataset. These are illustrative project targets, not Web Scraper defaults or universal thresholds.

Suppose a synthetic dataset contains 120 distinct, in-scope written reviews of a bottle model, all assessed under the same rule. Eighteen mention leakage: 18 / 120 × 100 = 15% of the analysed reviews.

Report the count, percentage, period, sources and collection rules together. The 15% measures leakage mentions among analysed reviews. It cannot establish a product defect rate because the denominator contains reviews rather than sold units, and the complaints do not prove a shared physical cause.

Compare consistent scopes over time. Adding a retailer, language or different review sort can change the observed complaint share without any change in the product.

Validate before scheduling collection

Download a pilot export and compare representative records with their source pages. Include exceptional records and products with no reviews.

Acceptance check	What should hold
Review identity	Source keys are unique in the current-review table; repeated observations remain separate
Product attribution	Reviewed subjects and observed page variants follow the source evidence
Field integrity	Ratings fit their original scales; dates retain their precision; fields belong to the same review
Text coverage	Full bodies match the source; confirmed rating-only records remain distinct from extraction failures
Collection coverage	Later batches are present, stopping reasons are explained and every input product has an outcome
Repeatability	Another run recognises existing reviews and preserves meaningful changes

Run the tested sitemap in Cloud with the intended execution settings before enabling a recurring schedule. Configure automated exports after checking delivery and ingestion. Automatic export sends the job dataset after completion; it does not merge review identities across jobs for you.

For a synthetic scheduling example, suppose one bottle model receives about 40 new reviews per week on one source. Use a Daily schedule at a chosen time and scheduler time zone, with the collection targeting newest-first reviews from the preceding 14 days. Consecutive daily runs overlap by 14 - 1 = 13 days, and the synthetic expected volume is 40 / 7 × 14 = 80 review rows per run before additional page appearances or syndicated copies.

Daily controls when the sitemap runs; it does not itself calculate a rolling publication-date cutoff. Use a bounded collection with enough headroom to reach the oldest required date, verify that coverage after each run and apply the 14-day date filter downstream. This can recover late-indexed reviews and edits within that collected window without revisiting the full history, while the upsert records repeated reviews as observations. Add a separate, less frequent schedule that revisits older reviews when changes outside the recent window matter.

Web Scraper's Data quality control checks record counts, field population and failed or empty-page proportions. Add downstream checks for semantic attribution and duplicate clusters. A job can complete while failing its configured data quality checks.

Run a clean pilot first, then use Fill with suggested values from that finished job as a starting point. Adjust for normal variation in the expected product set rather than treating one run as a fixed standard:

• Set Minimum record count for the expected product set and collection scope, allowing for ordinary changes rather than using one unusually large run as the baseline.
• Keep Maximum failed page % low enough to expose meaningful coverage gaps. CAPTCHA, 403 and 429 responses can leave affected review widgets or pages absent from an otherwise populated dataset.
• Allow genuine zero-review products when setting Maximum empty page %; a review-only page producing no records can be an expected empty result.
• Leave room for rating-only entries and missing titles when setting Minimum % of fields that must be filled, while retaining enough sensitivity to catch a text selector that stopped matching. Add a downstream text-status check so an allowed empty body does not conceal an extraction failure.

These are configuration guidelines, not fixed numbers or Web Scraper defaults. Check the resulting status in the sitemap's Data quality control tab and scraping jobs table, and enable Email or Cloud notifications when a failed check needs attention.

Before expanding the product list, ask a colleague to trace a leakage result back to its source text, product match, collection scope and classification rule.

Test your review sitemap locally, then start a seven-day Web Scraper Cloud trial to validate recurring collection.