ChatGPT Analysis: The Truth About Brand Visibility
▼ Summary
– Research shows AI responses are highly variable, with ChatGPT rarely recommending the same brands in the same order across multiple responses to the same prompt.
– In competitive B2B software categories, ChatGPT mentions about twice as many total brands as in niche categories, though each individual response typically names around 10 brands.
– Only about 5 brands, on average, achieve “dominant” status by being mentioned in 80%+ of responses, with established brands more likely to dominate, especially in competitive categories.
– Nuanced prompts (including a persona) do not significantly reduce the total number of brands mentioned but make it slightly harder for a brand to reach the dominant tier.
– For accurate AI visibility tracking, marketers should run key prompts multiple times (e.g., ~5 times) to gauge if their brand is in the long tail, middle, or dominant tier, rather than relying on a single check.
Understanding how artificial intelligence platforms like ChatGPT recommend brands is becoming a critical component of digital strategy. New research sheds light on the probabilistic nature of these systems, revealing significant implications for how businesses should approach tracking their visibility in AI-generated responses. The findings suggest that achieving consistent mention requires a nuanced strategy, particularly for brands not already holding dominant market positions.
Recent analysis builds upon foundational work by exploring how prompt structure and market competitiveness influence AI recommendations. The study focused exclusively on B2B software categories, testing twelve distinct prompts. Half targeted highly competitive markets like accounting software, while the other half addressed niche areas such as user entity behavior analytics software. For each category, researchers used both simple queries and nuanced prompts that included specific personas and use cases.
To gather robust data, each prompt was submitted one hundred times through the public version of ChatGPT, simulating different users with unique IP addresses for each interaction. This methodology aimed to capture the natural variability in AI responses. A key limitation is that the research centers on ChatGPT; however, the core principles likely apply to other large language models due to their shared probabilistic foundations.
The results highlight a dynamic landscape. On average, ChatGPT will mention 44 brands across 100 different responses to the same prompt, though this number fluctuates widely by category. In competitive markets, the AI draws from a pool roughly twice as large as it does for niche topics. Interestingly, while nuanced prompts sometimes yielded slightly fewer brand mentions, the effect was inconsistent. The more significant finding relates to consistency.
When asked for a recommendation 100 times, only about five brands, on average, are mentioned in over 80% of the responses. These dominant brands typically have strong, established recognition. For instance, in accounting software, names like QuickBooks, Xero, and FreshBooks consistently appear. This represents just 11% of the total brands the AI might mention across all responses, indicating fierce competition for top-of-mind status.
The strategic advantage shifts in less crowded markets. In niche categories, a remarkable 21% of all mentioned brands achieve dominant status, compared to a mere 7% in competitive fields. Here, the majority of brands in broad categories languish in a long tail, appearing in less than 20% of responses. Adding persona details to a prompt makes the winner’s circle slightly more exclusive but doesn’t drastically shrink the total number of brands the AI considers. This suggests the model may lack deep enough brand knowledge to make highly selective recommendations based on nuanced criteria.
For B2B marketers, these insights are actionable. If you’re not a dominant brand, pick your battles – niche down. Competing in a broad, established category against giants requires immense brand marketing resources. A more viable path is to differentiate sharply and own a specific segment. A company could aim to be recognized as “the best accounting software for commercial real estate companies,” a positioning ChatGPT is more likely to consistently reward.
The research also exposes a flaw in common tracking practices. Given the inherent inconsistency, checking a prompt’s output just once provides almost meaningless data. For statistically significant visibility scores, prompts need to be run dozens, if not hundreds, of times. A practical compromise for key bottom-of-funnel queries is to run each prompt approximately five times during a tracking cycle. This offers a reasonable indication of whether a brand appears consistently, occasionally, or rarely.
The goal should be to categorize performance into tiers: the long tail (under 20% visibility), the visible middle (20-80%), or the dominant tier (over 80%). This framework provides a data-driven way to assess competitive standing. Future research will delve deeper, examining how ChatGPT describes brands, whether different prompts with the same intent yield similar recommendations, and the consistency of brand ranking within individual responses.
(Source: Search Engine Land)
