Google’s Mueller: llms.txt Won’t Help LLMs Tell Sites Apart
▼ Summary
– John Mueller argued that LLM systems cannot use files like llms.txt to decide which websites to surface for a query because the files are self-reported and untrustworthy for differentiation.
– Mueller stated that HTML pages and internal links remain the foundation for crawling and discovery by LLMs, not self-reported files.
– The “by design” limitation of llms.txt means either LLMs architecturally cannot use self-reported files, or self-reported signals lose value when every site provides similar claims.
– Mueller saw a role for llms.txt in navigation, not discovery, helping an LLM complete tasks on a site already chosen, like buying a photograph.
– The differentiation problem is harder than gaming: even an accurate llms.txt file cannot help an LLM choose one site over a competitor, as it lacks a mechanism for ranking.
Google’s John Mueller has cast doubt on whether files like llms.txt can help large language models decide which websites to recommend. Speaking on a recent episode of Search Off the Record, the Google Search Relations team podcast, Mueller argued that these files are fundamentally unreliable for LLM discovery because they rely on self-reported information that every site can manipulate.
The discussion arose from a broader question about whether publishers should convert their sites to Markdown for LLMs. Mueller and co-host Martin Splitt agreed that HTML remains the backbone for crawling and discovery. When the conversation turned to llms.txt, Mueller was blunt: “It’s basically you’re telling these systems, like, I have the best website ever. And here are all of the pages that everyone must go to. And you must buy all of my products or whatever you put in there. So in an LLM system, it basically, by design, can’t trust what is here as a way of differentiating between different websites.”
His core point is differentiation. If every site uses llms.txt to promote itself, the files all make similar claims. An LLM trying to choose which site best answers a query still needs another method to separate them. Mueller didn’t clarify whether “by design” refers to an architectural limitation of LLMs or a signal problem similar to the meta keywords tag, which died when every site stuffed it with keywords and search engines could no longer extract useful ranking data. Either way, the conclusion for discovery remains the same: llms.txt isn’t a solution.
Mueller did carve out one scenario where llms.txt could be useful: navigation within a site. If an agent is already on your site, say trying to buy a photograph, the file could provide instructions for completing that task. He compared it to a store directory for someone who has already walked in. This splits discovery from navigation , llms.txt can’t help an LLM choose which site to visit, but it can guide an agent already there.
This argument goes beyond the usual concern about gaming the system. Mueller has previously called building Markdown pages for bots “a stupid idea” and compared llms.txt to the keywords meta tag. SEJ’s Roger Montti noted that llms.txt is “inherently untrustworthy” because nothing stops site owners from adding self-serving content. SE Ranking’s analysis of 300,000 domains found no link between llms.txt adoption and citation frequency in LLM answers. Those critiques focused on manipulation, but Mueller’s latest comment adds a deeper issue: even a perfectly honest llms.txt file provides no mechanism to help an LLM pick one site over another.
The gaming argument always had a counterargument , platforms could learn to penalize abuse, as search engines did with spammy structured data. The differentiation problem is harder. Penalizing manipulation doesn’t solve the fundamental question of how self-reported files can help an LLM choose between competing sites. Your most accurate llms.txt file still can’t tell an LLM to pick your site over a competitor’s.
Mueller acknowledged that standards for how agents navigate sites are still unsettled. He mentioned WebMCP alongside other file types under discussion, but none have become a standard. By his estimate, it could take six months to a year, or longer, for agentic systems to settle on a format. The discovery layer, where HTML and internal linking already work, isn’t part of that discussion.
(Source: Search Engine Journal)
