Study: 80% of ChatGPT product picks shift with search enabled
▼ Summary
– Enabling search in ChatGPT changed 80.2% of product recommendations compared to when search was disabled, based on 20,000 responses.
– Only 19.8% of products recommended without search also appeared in recommendations generated with search enabled.
– Products that appeared in 100% of search-disabled responses had a low 15.8% overlap with search-enabled responses.
– Responses with search enabled averaged 5.2 products per prompt, versus 6.2 products without search.
– The study found a 0.4 Pearson correlation between how often a product was mentioned in cited sources and its recommendation frequency, but did not establish causation.
A new study analyzing 20,000 responses reveals that ChatGPT’s product recommendations shift by 80.2% when search is enabled, according to research by Jeff Oxford, founder and CEO of Visibility Labs.
Oxford tested 1,000 distinct product-recommendation prompts, running each one ten times with ChatGPT search enabled and ten times with search disabled. The results show that only 19.8% of products recommended without search also appeared in responses generated when search was active.
The impact on top picks was significant. Even the products ChatGPT recommended most consistently without search rarely carried over. Among items appearing in 100% of search-disabled responses, just 15.8% remained when search was turned on. Oxford had expected the most consistently recommended products to stay common with search enabled, but that group instead showed the lowest overlap of any category.
The study also tracked whether products mentioned in ChatGPT’s cited sources appeared more frequently in its recommendations. Oxford found a 0.4 Pearson correlation between cited-source mentions and recommendation frequency. He measured this using a “Visibility Score,” defined as the percentage of runs in which a product appeared for a given prompt. Products mentioned more often in cited sources tended to earn higher Visibility Scores. However, the analysis did not establish that cited-source mentions caused products to be recommended.
Search also narrowed the scope of recommendations. ChatGPT responses with search enabled contained an average of 5.2 products, compared with 6.2 when search was disabled. Across ten runs of each prompt, ChatGPT returned an average of 19 unique products per prompt with search enabled and 21.8 with search disabled.
Why this matters for marketers and SEO professionals. Search fundamentally changed which products ChatGPT recommended, including items it named every time when web access was disabled. The findings suggest that products appearing in cited sources may receive greater visibility when search is enabled, though the study does not determine whether cited-source visibility matters more than broader web visibility.
About the data. Oxford analyzed 1,000 product-recommendation prompts, running each ten times with search enabled and ten times with search disabled. Product names were standardized so naming variations counted as the same product. Because the study was observational, it did not establish a causal relationship between cited-source mentions and recommendation frequency.
The full report is titled: ChatGPT’s Product Recommendations Change 80.2% When Search is Enabled vs Disabled (Study of 20,000 Responses). For further context, earlier research showed that AI recommendation lists repeat less than 1% of the time.
(Source: Search Engine Land)
