Google & Bing: Skip Separate LLM Markdown Pages
▼ Summary
– Google and Bing representatives advise against creating separate markdown pages for LLMs, as serving different content to bots and users may be considered cloaking.
– Google’s John Mueller states LLMs are trained on standard HTML and questions the need for a non-user-facing markdown version.
– Mueller has also criticized the idea as unnecessary, noting LLMs can process various formats like images and HTML effectively.
– Bing’s Fabrice Canel warns that maintaining separate bot versions increases crawl load and risks content neglect or breakage.
– The overall recommendation is to avoid shortcuts like separate markdown files, as they are often temporary and can negatively impact SEO.
Creating separate markdown pages specifically for large language models is a practice that search engines are actively discouraging. Representatives from both Google and Bing have explicitly advised against this tactic, warning that it could be interpreted as a form of cloaking. Cloaking, which involves showing different content to users and search engine crawlers, is a direct violation of Google’s Webmaster Guidelines and can lead to severe penalties. The core recommendation is to serve a single, high-quality version of your content that is accessible and valuable to both human visitors and automated systems.
The discussion gained public attention when SEO expert Lily Ray posed a question on Bluesky, asking about the growing trend of creating separate markdown or JSON pages to serve to LLM bots. The responses from major search engines were swift and clear. Google’s Search Advocate, John Mueller, expressed skepticism, stating he was unaware of any need for such separate pages. He pointed out that LLMs have been trained on standard HTML web pages from their inception and are fully capable of parsing them. Mueller questioned the logic of creating a page version that no user would ever see, emphasizing that if equivalence checking is necessary, the original HTML should suffice.
In stronger terms, Mueller has previously criticized the concept as fundamentally flawed, using hyperbole to make his point. He suggested that if the goal is to create a different format for machines, one might as well convert an entire website into an image, an obviously impractical idea meant to highlight the absurdity of over-engineering for crawlers. His consistent message is that focusing on clean, well-structured HTML is the most reliable path.
The perspective from Microsoft Bing aligns closely. Fabrice Canel from Bing responded by questioning the efficiency of doubling the crawl load for search engines. He noted that Bing’s crawlers would inevitably crawl the standard version to check for similarity anyway. Canel highlighted a practical issue: non-user-facing versions of content, like crawlable AJAX, are often neglected and can break over time because they lack the human oversight that comes with user-visible pages. His advice reinforces a principle of modern SEO: “Less is more.” Bing advocates for enriching standard web pages with structured data (Schema) and trusts in their AI’s advanced ability to understand standard web content.
The underlying concern for search marketers is significant. In an era where new AI search experiences are emerging, there is a temptation to find technical shortcuts to gain an edge. However, as Lily Ray noted on LinkedIn, such shortcuts typically offer only temporary benefits at best. At worst, they can backfire spectacularly, damaging a site’s standing in organic search results. The consensus from search engine authorities is unambiguous: investing in a unified, user-first content strategy is not only safer but is the most sustainable approach for long-term visibility across all search interfaces, including those powered by advanced AI.
(Source: Search Engine Land)
