Scraping ChatGPT and Perplexity: why answers matter
October 09, 2026
market research, data quality, AI search, web scraping, Competitive intelligence
Businesses are scraping ChatGPT and Perplexity to study the answers they produce: which products they recommend, how they describe companies and which pages they cite. AI services have become another source of competitive intelligence, alongside the websites they help people discover.
What the ranking shows
In Decodo's 2026 annual ranking, ChatGPT appears third and Perplexity fifth. Both are new entrants in that report's top ten, which is based on anonymised usage data from Decodo's users. The report page was updated on 10 September 2026, and TechRadar Pro covered the findings on 5 October.
These are positions within one provider's customer activity. They do not establish an internet-wide scraping ranking, and a first appearance in this list does not tell us when scraping either service began.
Decodo describes users collecting response text, brand mentions and citations. Our interpretation is that generated answers are becoming an additional layer of market evidence: businesses want to observe how an AI service presents information after it has selected and summarised it.
Why businesses are scraping ChatGPT and Perplexity
A product catalogue shows what a seller publishes. A collected AI answer shows how a service presents products in response to a particular question.
Consider a hypothetical outdoor retailer investigating the prompt, “Which waterproof hiking jackets under $150 are worth buying in the US?”
The retailer could check whether its jackets appear, which alternatives are recommended, whether quoted prices are current and whether the answer cites a manufacturer, retailer or review site. A missing product and an incorrect price would prompt different investigations: visibility in the tested answers versus accuracy of the information presented.
A competitive intelligence project might submit its own questions and retain the returned answers and source links. This creates records of the service's output under the conditions tested. The commercial value comes from identifying specific recommendations, omissions or descriptions worth investigating.
Mentions, citations and recommendations answer different questions
Keep these signals separate when interpreting the results:
| Signal | What an observation establishes | What needs separate evidence |
|---|---|---|
| Brand mention | The answer named the business or product | Whether the description was favourable |
| Recommendation | The answer presented an option as suitable for the question | Whether other questions or users receive that recommendation |
| Citation | The answer linked to a particular source | Whether it recommended that source's owner |
| Referral visit | Website analytics attributed a visit to the service | How often the brand appeared without generating a visit |
An answer might recommend a competitor while citing your comparison article. That is a citation for your page and a recommendation for somebody else. Combining them into one visibility score would obscure what happened.
Why the interface matters
Official APIs provide a programmatic way to generate answers with sources. OpenAI's web search API returns answer text and citation annotations. However, an API observation needs to be labelled by the interface and configuration used. It should not automatically be presented as a recording of the consumer website.
Perplexity's own API FAQ says its API and website use the same search system, but outputs may differ because their configurations or underlying models can differ. OpenAI also documents that ChatGPT search may use saved memories when rewriting queries and location information to improve relevance. These are reasons to define the experience being measured before comparing results.
The access route also affects permission. OpenAI's individual Europe terms prohibit automated or programmatic extraction of data or output. Perplexity's standard user terms restrict scraping and collection, subject to their stated written-permission and applicable-law exceptions. Developer APIs have separate terms. A report showing that collection occurs does not establish that every method is permitted.
What makes an answer dataset useful
Preserve the exact question, collection time, service, interface, relevant market and available model or search settings alongside the full answer and cited URLs. That context lets an analyst distinguish an observed change from a change in the way the test was run.
Repeat questions under comparable conditions before describing a pattern. OpenAI's API documentation explains that model outputs vary and behaviour can change between model versions. Keep failed requests and incomplete responses separate from valid answers that omit a brand.
For a hypothetical result, “our jacket appeared in six of ten test answers” describes the observations. It does not establish that 60% of the platform's users saw it. The prompt set and collection conditions define the scope of the finding; audience reach requires different evidence.
Check AI claims against the original pages
Return to the jacket example. Suppose an answer recommends a product at a price the retailer believes is outdated. Check the exact model, variant, currency and cited page. The source could describe an old promotion or a different product, or the answer could have misinterpreted it.
Keep a source dataset containing the product identifier, published price, availability, source URL and observation date. This is where market research from public websites complements answer monitoring: it supplies evidence against which a generated claim can be checked.
Review the cited page itself before attributing an error to it. A citation does not prove that every statement in an answer came from that page. Correcting outdated source material improves that source; it does not guarantee a particular future AI response.
Web Scraper supports collecting these source pages across accessible product listings, company sites and directories. Start with a representative sample, build and test a sitemap in the browser extension, then schedule recurring collection in Web Scraper Cloud once the extracted fields are correct.
The development highlighted by the report expands the questions a business can investigate. Alongside asking what websites publish, it can examine how an answer engine presents those offerings. Combining the recorded answer with current source evidence turns an unexplained visibility result into a finding a team can act on.