Cloudflare AI Bot Blocking May Also Stop Googlebot Indexing
▼ Summary
– A Redditor reported that enabling Cloudflare’s AI Training block causes Googlebot and Bingbot to receive HTTP 403 errors when fetching their sitemap.
– The issue occurs because Cloudflare now classifies Googlebot and Bingbot as “Search + Training” bots, so blocking AI training also blocks these search crawlers.
– Google’s John Mueller responded to the Redditor, requesting a direct message to investigate the issue further.
– Cloudflare’s official announcement states that starting September 15, 2026, mixed-purpose crawlers combining Search and Training will be blocked by all configurations that block AI training, including the legacy “Block AI bots” option.
– The Redditor’s report of the blocking occurring now is curious, as the change is officially scheduled to take effect in September.
Blocking AI crawlers through Cloudflare may come with an unintended consequence: it could also block legitimate search engine bots like Googlebot and Bingbot from indexing a site. A new report suggests this is already happening for some users, even though Cloudflare’s official policy change isn’t scheduled until September. Whether this is a glitch or a configuration issue remains unclear, but the incident highlights the need for site owners to review their settings before the deadline.
A Redditor in the r/SEO community recently raised the issue, asking whether others had encountered Cloudflare blocking Googlebot under the guise of AI prevention. The user explained that enabling the “AI Training = Block” setting caused both Googlebot and Bingbot to receive HTTP 403 responses when attempting to fetch their sitemap. Disabling the block restored access immediately. The user noted that Cloudflare appears to classify these search engines as “Search + Training” bots, which means the training block inadvertently impacts them as well. The dilemma is straightforward: allow AI training to keep search engines happy, or block AI training and risk losing indexing altogether.
John Mueller from Google responded to the thread, asking the original poster to send a direct message so he could investigate further. Another commenter speculated that Cloudflare might be blocking fake bots spoofing Googlebot’s user agent. The original poster clarified that this wasn’t the case. They explained that Cloudflare’s own dashboard shows Googlebot and Bingbot as blocked when Bot Fight Mode is enabled, and that sitemap fetches return 403 errors until the mode is turned off. The user questioned whether Bot Fight Mode is interfering with legitimate crawlers or whether the dashboard is misreporting the situation.
Cloudflare’s official documentation confirms that starting September 15, 2026, the company will enforce updated defaults for new domains. Bots classified as Training or Agent will be blocked on pages displaying ads, while Search remains allowed. Crucially, mixed-purpose crawlers that combine Search and Training will also be blocked under any configuration that blocks AI training, including the legacy “Block AI bots” option. Until that date, customers can opt out of these new defaults.
Given that the change is supposed to take effect in September, it’s surprising to see reports of this behavior occurring now. Site owners who rely on organic search traffic should take this as a warning to audit their Cloudflare settings sooner rather than later. Waiting until the official rollout could leave your site invisible to search engines, and recovering from that kind of indexing loss takes time.
(Source: Search Engine Journal)
