How to Monitor MAP Violations With Web Scraping
September 07, 2026
price monitoring, e-commerce data, web scraping
To monitor minimum advertised price (MAP) violations with web scraping, collect seller- and variant-level advertised-price observations on a schedule, match them to the policy threshold that was effective at the observation time, and send only eligible below-threshold records for review.
The important word is review. A scraper can record what a page displayed. It cannot decide whether the product and seller were matched correctly, whether a coupon or basket price falls within your policy, or whether an observation is legally actionable. A reliable workflow keeps extraction, classification and confirmation separate.
Treat MAP monitoring as three separate states
A basic monitor appears simple: scrape a price, compare it with a threshold and send an alert when the price is lower. That model fails when a page contains a list price, a sale price, a coupon, several variants and offers from multiple marketplace sellers.
Use three states instead:
- Observation: what a particular page state displayed, where and when.
- Candidate flag: an observation that passed product, seller, price-state and threshold checks.
- Confirmed issue: the outcome assigned through the organisation's approved review process.
A dismissed flag must not overwrite its observation. Product mismatches, ineligible prices and stale pages are different outcomes.
Implementation follows five stages: define scope, extract observations, match and normalise them, apply supplied rules, and route results for review. Web Scraper provides extraction and delivery; the customer owns scope, master data, policy rules, legal assessment and enforcement. The MAP monitoring solution page gives the shorter product-level overview.
1. Define the policy and monitoring contract first
Start with the decision the review team must make, not a selector for .price.
For each policy rule, define:
- the product or variant it covers;
- the market, currency, channel and seller types in scope;
- the advertised price states eligible for comparison;
- promotion, launch-period, seller or channel exceptions;
- the threshold amount and its effective period; and
- who may confirm, dismiss or escalate a candidate flag.
Define “advertised” operationally, including the treatment of sale, reference, member, coupon, basket, bundle and loyalty prices. Store that interpretation as versioned policy data, not page selectors.
Legal review belongs here. The US Federal Trade Commission's guidance discusses MAP programmes with competition-law qualifications, while the European Commission's inquiry addresses e-commerce restrictions in the EU. Obtain market-specific advice: scraping supplies observations, not legal conclusions.
For every source pattern, record the domain, page type, expected market and currency, products, identifiers, required interactions and fields, maximum observation age and source owner.
“Monitor 500 products” is not a measurable contract. “Attempt every priority product-market-source combination every six hours and preserve each marketplace seller offer separately” is.
2. Build canonical product and seller records
Family titles such as “X300 headphones” are unsafe keys when colours, pack sizes, conditions or generations differ.
A canonical record should contain an internal product_id, brand, model, relevant variant attributes, market and source aliases. Use a GTIN when exposed, but keep MPNs, retailer SKUs and marketplace identifiers in separate fields.
Use an explicit matching order:
- an approved source-to-product mapping;
- an exact GTIN or another governed product identifier;
- an exact MPN plus brand;
- an approved retailer SKU mapping;
- constrained attributes such as model, capacity, colour, condition and pack size; and
- manual review when ambiguity remains.
Store matched_product_id, match_method, match_confidence and match_status. Never let a title match cross a variant boundary silently; ambiguous records go to a mapping queue.
For sellers, preserve the display name, source ID and storefront URL before canonical matching. Identity and authorisation are separate: the first is a match, while the second is an effective-dated business rule.
3. Store one immutable row per offer observation
Do not begin with a violations table. Begin with observations that can be re-evaluated if mappings or rules change.
| Field group | Recommended fields |
|---|---|
| Observation | observation_id, observed_at_utc, job_id, source_id |
| Provenance | requested URL, final URL, page type, market, extraction version |
| Product | source product ID, title, GTIN, MPN, variant attributes, condition |
| Seller | source seller ID, storefront URL, displayed seller name |
| Price | raw price text, parsed amount, currency, price state, reference price |
| Conditions | sale label, coupon text, membership requirement, basket requirement, bundle contents |
| Availability | stock state, offer visibility and relevant fulfilment condition |
| Validation | page identity, required-field status and source-batch quality result |
Keep raw and parsed values together. $1,299.00, 1.299,00 € and from 1,299 need different parsing. If only 1299 survives, locale and qualifier errors become harder to diagnose.
Use one observation per seller offer. Keep seller, price, condition and promotion selectors inside the same repeating offer wrapper so missing fields cannot misalign independent lists.
Record time in an unambiguous, offset-aware format such as 2026-09-07T08:15:22Z, following RFC 3339. The final URL is also important: a requested product page may redirect to a region selector, replacement item or access page.
Create an idempotency key from source, product, variant, seller, page state and observation window. Retries can then deduplicate an event without overwriting its raw observation.
4. Classify price states before comparing amounts
Selecting the lowest number on the page is not a safe shortcut. First classify the visible price state, then let the applicable policy rule decide whether it is eligible.
| Price state | Evidence to retain | Evaluation treatment |
|---|---|---|
| Regular advertised price | Current displayed amount and currency | Evaluate when included by the policy |
| Sale price | Sale amount, reference amount and sale label | Apply the approved sale rule |
| Coupon price | Displayed amount, coupon value and conditions | Keep separate until coupon eligibility is defined |
| Member or loyalty price | Amount and membership condition | Do not treat as an ordinary price by default |
| Basket-revealed price | Pre-basket display and revealed amount, if collection is permitted | Apply the approved basket-price rule |
| Bundle or “from” price | Total, included products or visible qualifier | Evaluate only after equivalence is established |
| Marketplace offer | Seller, item condition and offer amount | Match seller and exact item before evaluation |
| Unknown | Raw text and page context | Quarantine rather than guess |
The classifier should return fields such as price_state, comparison_eligible, eligibility_reason and rule_version. A selector can report that the page said “$15 coupon at checkout”. The customer-owned policy rule decides what that means.
A rise in unknown states can reveal a new promotion or page component before incorrect flags spread.
5. Version thresholds, exceptions and evaluation rules
A single map_price column loses history when a threshold changes. Use an effective-dated table instead.
| Field | Purpose |
|---|---|
threshold_rule_id |
Stable identifier for the rule applied |
product_id and variant_id |
Exact item governed by the rule |
market, channel, currency |
Context in which the threshold is valid |
threshold_amount |
Amount supplied by the policy owner |
effective_from, effective_to |
Rule's valid interval |
eligible_price_states |
Price classifications allowed into comparison |
exception_id |
Approved campaign, seller or channel exception |
policy_version |
Reconstructable version of the approved logic |
An observation from 30 September must use the rule valid then, even if the threshold changes on 1 October. Applying today's amount to history can manufacture or erase flags.
Use a fixed evaluation order:
- Did the intended page and offer pass source-quality checks?
- Is the exact product and variant match accepted?
- Is the seller sufficiently resolved?
- Is the price state eligible under the applicable rule version?
- Is an effective threshold available for the market, channel, currency and observation time?
- Does an approved exception cover the observation?
- Is the eligible advertised amount below the supplied threshold?
Only the final result creates a pending_review candidate. Preserve non-flag reason codes such as COUPON_EXCLUDED, VARIANT_UNMATCHED, NO_EFFECTIVE_THRESHOLD, CURRENCY_MISMATCH and SOURCE_QUALITY_FAILED. A blank result cannot explain which part of the workflow failed.
6. Build and test the extraction layer
Select a deliberately varied sample before scaling: a normal retailer page, a discounted item, an unavailable product, a marketplace seller list, a JavaScript-rendered price and a product with variants.
Build and test each source-specific sitemap with the Web Scraper browser extension. Use Element Preview and Data Preview across several products and layouts, then inspect a limited export before automation.
Use Fast when the returned HTML contains the data and no interaction is needed. Use the FullJS driver for JavaScript, scrolling, clicks or Website State Setup. Test that each selected variant remains active during extraction.
For each source template, verify:
- ordinary, promotional and unavailable layouts;
- later pagination and expanded seller-offer states;
- intended market, currency and location state;
- behaviour when optional fields are absent;
- one correctly paired record per seller offer; and
- what an access page, redirect or changed template looks like in the output.
If a page returns successfully but the expected price or seller is missing, inspect the returned content before changing delays or proxies. A 200 OK response does not prove the intended page or data was obtained, as the guide to diagnosing successful responses with no useful data explains.
7. Schedule according to detection risk
There is no universal monitoring frequency. Set a maximum acceptable observation age for each product-source tier.
- High-risk launches may need several checks per day.
- Stable authorised-dealer pages may need only daily or weekly checks.
- Marketplace offer lists may justify shorter intervals.
- A follow-up observation can test whether a candidate persists.
Measure expected observations, not merely rows. Without a product-source denominator, 12,000 collected prices can hide missing priority checks.
Stagger sources rather than starting every job at the same minute. Retain both the scheduled window and the actual observation time. Configure the Cloud Scheduler only after the same sitemap, driver, proxy and delay settings have passed a representative test. If another system orchestrates the workflow, the Cloud API can launch an existing sitemap and retrieve completed data; it is not a generic URL-in, dataset-out endpoint.
Respect applicable access rules, terms, privacy requirements and collection limits. RFC 9309 standardises robots.txt but does not make it access authorisation.
8. Gate data quality before creating flags
A scraping job can finish while producing an unusable MAP dataset. Monitor both extraction health and downstream rule readiness.
| Metric | What it reveals |
|---|---|
| Expected-observation coverage | Product-source checks that never occurred |
| Page acceptance rate | Redirects, access pages or incorrect page types |
| Required-field fill rate | Broken price, seller, currency or variant selectors |
| Product and seller match rate | Identity gaps that prevent comparison |
| Eligible-state rate | Observations that current rules can evaluate |
| Unknown-state rate | New promotions or layout patterns |
| Freshness compliance | Observations arriving too late for the decision |
| Duplicate rate | Retries or navigation inflating the dataset |
| Flag and dismissal rates | Matching errors, rule drift or genuine changes |
Trend metrics by source and template. Normal row counts can hide a redesigned seller component that leaves seller IDs blank.
Use Web Scraper Cloud data-quality controls for extraction checks such as minimum record count, maximum failed or empty page percentages and minimum field population. Add downstream gates for product matching, seller matching, price-state eligibility, threshold coverage and historical anomalies.
Quarantine only the suspect source batch. Healthy retailer batches can proceed while the affected marketplace is repaired. “No candidate flags today” is meaningful only when the dashboard also shows what was not observed or could not be evaluated.
9. Build an evidence package and reason-based review queue
Attach or reference the raw and parsed price, visible price conditions, product and variant match, seller identity, requested and final URLs, market, currency, observation time, threshold and policy versions, extraction version and source-quality result.
If the approved evidence standard requires screenshots, saved HTML, cryptographic hashes or another immutable artefact, add a dedicated downstream capture and retention process. Do not assume a scraped dataset or live URL alone satisfies legal, contractual or internal evidence requirements.
Route records by reason and confidence:
- high-confidence product, seller and eligible-price match;
- product or variant match needs confirmation;
- seller identity is unresolved;
- price state or policy eligibility is unclear;
- threshold or currency is missing;
- source evidence failed quality checks; or
- a previously reviewed issue has reappeared.
Useful reviewer outcomes include confirmed_issue, dismissed_product_mismatch, dismissed_ineligible_price, approved_exception, needs_recapture and needs_legal_review. Preserve the reviewer, time and reason. Feed dismissals into mapping rules and extraction tests rather than deleting them.
Escalation remains a separate workflow. It should use approved authorisation, policy and legal processes instead of firing automatically because one number was below a threshold.
Worked example: four low-looking prices, one candidate
Assume a brand monitors new black X300 headphones in the US. Threshold rule MAP-US-X300-BLK-004 is $199 and applies to public regular and sale prices throughout September. The example is hypothetical.
| Observation | Classification and match | Outcome |
|---|---|---|
| Retailer A shows the black X300 at a $189 sale price with no exception | Exact MPN, eligible sale state, correct market and rule period | Candidate flag, $10 below threshold |
| Retailer B displays $209 plus “$15 coupon at checkout” | Display remains $209; coupon state is stored separately and excluded by this rule | No flag, COUPON_EXCLUDED |
| Marketplace seller C lists “X300 black refurbished” at $195 | Condition does not match the canonical new-product record | Product-match rejection |
| Marketplace seller D lists the white X300 at $179 | Variant does not match the black threshold record | Mapping review, no comparison |
The workflow creates one pending candidate, not four violations. Its evidence package preserves the seller, product match, price evidence, URL, time, rule version and quality result for review.
Now suppose the same scheduled batch includes a marketplace whose redesign removes seller IDs from most offers. Record counts remain normal, but the source-level seller fill-rate gate fails. That marketplace batch is quarantined before comparison while healthy retailer batches continue. Extraction quality and policy logic are separate controls, and both must pass.
Common failure modes to design out
| Failure | Consequence | Control |
|---|---|---|
One generic price field |
Sale, reference, coupon and basket prices are mixed | Store price type, label and conditions separately |
| Product title used as the only key | Wrong pack, condition or variant is compared | Use canonical identifiers and a mapping queue |
| Marketplace lead offer treated as the seller | Offer attributed to the wrong merchant | Preserve one record per seller offer |
| Current threshold applied to history | Past outcomes change when policy data changes | Version thresholds and join by observation time |
| Every retry creates another alert | Review queue fills with duplicates | Use stable idempotency and alert-state keys |
| Missing sources interpreted as compliant | Coverage failure looks like fewer issues | Publish coverage and freshness with flag counts |
| Unknown promotion forced into a known class | Incorrect eligibility decision | Retain raw labels and quarantine unknown states |
Roll out in controlled stages
Begin with a narrow but difficult pilot, such as ten products across three source patterns. Include multiple variants, a seller-offer list, a coupon, a sale and a known exception. Establish the schema, mappings, threshold history, quality gates and review outcomes before expanding.
Run it in shadow mode beside manual checks. Classify disagreements by timing, market state, variant, seller expansion, extraction or interpretation instead of assuming the manual result is always correct.
Expand only when you can reconstruct every candidate from its raw observation, product and seller matches, price classification, effective threshold and quality results. Confirm that a failed source can be isolated without blocking healthy ones and that unknown states remain visible.
Where Web Scraper fits
| Workflow component | Web Scraper role | Customer-owned role |
|---|---|---|
| Source configuration | Build and test source-specific sitemaps in the browser extension | Choose permitted sources, products and page states |
| Extraction | Collect configured product, seller, price and promotion fields | Define canonical records and required evidence |
| Automation | Run tested sitemaps on schedules in Web Scraper Cloud | Set cadence, coverage targets and source priority |
| Data quality | Check extraction-level records, page outcomes and field population | Apply matching, threshold and business-rule gates |
| Delivery | Export completed datasets or connect completed jobs downstream | Store history, deduplicate events and manage review |
| Decision and action | None | Confirm issues, obtain legal review and enforce policy |
This suits teams that want control over product data and policy logic without operating recurring extraction infrastructure. Web Scraper is not a managed seller-identification, policy, case-management or enforcement service, and it is a poor fit for inaccessible or untestable sources.
Frequently asked questions
Can web scraping automatically confirm a MAP violation?
No. It can collect observations and support supplied comparison rules. Product identity, seller identity, price eligibility, exceptions, policy scope and legal interpretation still require customer-owned logic and, where appropriate, human review.
Which price should a MAP scraper collect?
Collect every relevant visible price state with its label and conditions: regular, sale, reference, coupon, member, basket, bundle and seller-offer prices. Classify them separately, then let the applicable policy rule choose the eligible amount. Do not select the lowest number without context.
Start with a representative set of difficult products and sources, then prove that each completed job produces traceable, correctly matched observations before increasing coverage.