AI chatbots clash on top software picks in 1/3 of categories

GetIntel released its AI Software Index on Wednesday, a study that put roughly 80 buyer-style questions to both ChatGPT and Gemini across 126 software categories. The exercise covered 1,825 brands and yielded 9,978 usable responses, all gathered on a single day, 6 August. Notably, the researchers used the live consumer apps rather than developer APIs.

The headline finding reads like a coin toss. In one out of every three categories, the two chatbots name a different top pick. Whether a buyer happens to open ChatGPT or Gemini can determine which company gets the recommendation. That level of divergence has serious implications for software vendors tracking their visibility.

The review sites have been skipped

What the assistants actually cite tells its own story. The most-cited domain across the entire study was techradar.com, appearing 1,114 times. Reddit came in second with 807 citations, and Zapier took third with 418. The traditional review platforms, G2, Capterra and TrustRadius, together accounted for just 5% of all citations.

That represents an industry being routed around. Those three sites were built to serve as the reference layer for software purchasing decisions. Now a consumer tech publication and a discussion forum are doing that job instead. Gartner managed 238 citations, fewer than Zapier.

Reddit holding second place is its own complication. The platform has been actively cracking down on brands seeding fake opinions for chatbots to repeat. It reports catching 25,000 such posts per day. The assistants are leaning hardest on the source that is hardest to keep clean.

ChatGPT sends traffic. Gemini does not

The two systems behave differently once they settle on a winner. When ChatGPT recommends a product, it cites that product’s own website 53% of the time. Gemini does so in only 13% of cases, preferring third-party pages instead.

That means the same recommendation carries different weight. Winning inside ChatGPT delivers a clickable link. Winning inside Gemini delivers a mention. For anyone still measuring publisher traffic as the outcome, that distinction matters.

Gemini’s independent sources get stranger. Its most-cited third-party pages are blog posts written by other software companies. The study notes these posts often cover categories unrelated to the publishing company’s own business.

The answer changes with who is asking, and where

Stating who you are shifts the result. In some categories, a brand goes from winning 34% of answers to winning 90%. The only variable is whether the buyer says “I’m a freelancer” or “we’re an enterprise.” Same category, same question, different customer.

Geography moves the needle too, and this is the finding a European reader should sit with. The number one recommendation flips between US-phrased and UK-phrased versions of the same question. That happened in roughly half the categories the study could measure. Regulators have spent two years debating whether AI answers opt out fairly. Nobody has asked whether these systems answer the same question the same way on both sides of the Atlantic.

Showing up and winning are not the same

The leaderboard data is unkind to the obvious names. HubSpot appears on more category leaderboards than anyone, 19 in total, and wins three. Salesforce appears on 13 and wins none at all. QuickBooks appears on seven and wins five.

Some categories are already settled. GitHub is the first recommendation in 91% of CI/CD answers and 72% of code review answers. Shopify takes 84% of e-commerce and Miro 83% of whiteboards. Square holds 78% of point of sale and Loom 76% of screen recording.

Others are wide open. In affiliate marketing software, the top pick wins only 5% of answers. Contract management sits at 13%. Invoicing, live chat and order fulfilment all sit at 19%. The winner there changes almost every time the question is reworded.

Who measured this, and why

The caveat is the whole business model. GetIntel is a Bengaluru company that sells AI visibility software. Its full index concludes that companies do not know whether AI names them. That is also its sales pitch. Founder Tarang Agarwal says so nearly outright.

Agarwal puts the problem plainly. A company can quote its Google ranking to the decimal, he said. It may have no idea whether ChatGPT has ever named it. “Closing that gap is the entire reason an AI visibility tool exists.”

It is a crowded pitch. Berlin’s Peec AI reached a $200m valuation selling the same promise. The practical work of earning citations still looks a lot like ranking on Google.

The method has real limits and they are worth stating. This is one day of data from two engines, with no error bars and no peer review. “Top pick” simply means the first brand named in an answer, which is a blunt measure of a nuanced reply. A single snapshot cannot show whether any of this is stable.

Which is the test worth setting. GetIntel says it will refresh the Index quarterly. If the disagreement rate is still near a third in November, buyers really are getting different answers from different machines. If it collapses, this was a photograph of a moving object.