Cloudflare’s New Robots.txt Generator Has a Strong Case
▼ Summary
– The author previously highlighted discrepancies between their robots.txt file and actual bot enforcement policies on their website.
– Cloudflare introduced Bot Preference Sync, a tool that automatically aligns robots.txt entries with dashboard settings to prevent contradictions.
– Despite the announcement, the feature had not been implemented in documentation or user files as of mid-September.
– The author criticizes the product for defaulting vendor-controlled decisions that may not accurately reflect specific website needs.
– Conflicting signals between text-based rules and edge enforcement can provide legal grounds for crawlers to ignore restrictions.
Cloudflare’s new Bot Preference Sync tool automates robots.txt generation, aiming to eliminate the dangerous gap between a site’s stated crawl preferences and its actual enforcement rules. While the feature promises to keep your robots.txt file aligned with your dashboard settings, it raises significant concerns about vendor-controlled policy defaults and the loss of granular control over AI crawler access.
The Problem of Drifting Policies
The core issue this product addresses is the inconsistency that often develops between a website’s robots.txt file and the blocking rules enforced at the network edge. A text file written months ago may no longer reflect current business decisions, while dashboard changes made recently might not be reflected in the public-facing crawl directives. This disconnect can signal to sophisticated crawlers that a site’s preferences are unreliable, potentially encouraging them to ignore restrictions or bypass enforcement measures.
As the author noted in an earlier analysis: “On August 20, I published a page arguing that a website’s robots.txt and its actual enforcement are two different things… My robots.txt had spent months welcoming Bytespider by name, long after I would have chosen otherwise.” Cloudflare’s new solution attempts to resolve this by automatically generating robots.txt entries based on user-defined policies in the Cloudflare dashboard.
How Bot Preference Sync Works
The feature operates by inserting a specific block into the robots.txt file, bounded by `# BEGIN Cloudflare Bot Preference Sync` and `# END Cloudflare Bot Preference Sync` markers. This block contains directives derived from three primary categories configured in the Security Settings: Search, Agent, and Training.
Users can choose to allow or block these categories across all pages or only on pages containing ads. For the Training category, selecting “Disallow” adds a `no-training` directive to the file. Crucially, the system maps individual bots to these categories using Cloudflare’s internal tracked bot list. This means users cannot exclude a specific crawler from the sync; if they desire finer control, they must disable the synchronization entirely and manage the robots.txt file manually.
Limitations in Granular Control
The reliance on broad categories rather than individual bot policies creates friction for publishers who make nuanced decisions based on each AI company’s value proposition. The author explains their own strategy:
“My own crawler policy does not fit any of Bot Preference Sync’s three categories. I allow OpenAI’s GPTBot, Anthropic’s crawler and PerplexityBot. I block Bytespider and meta-externalagent. Every one of those companies trains models. My rule is a question I ask per company: What am I getting in return? The first three put my pages in front of people who ask assistants questions. The other two take and return nothing.”
With Bot Preference Sync, such a tailored approach is impossible without turning off the automation. Setting Training to disallow would inadvertently block desirable partners like OpenAI, while allowing it would fail to distinguish between beneficial and non-beneficial trainers at the file level, even if the edge firewall blocks the latter.
The Four Conditions for Opaque Crawlers
A critical aspect of the Training category is how Cloudflare defines “opaque” AI crawlers. If a publisher sets Training to disallow, Cloudflare will block any AI crawler that fails to meet four specific disclosure conditions. These conditions are designed to ensure transparency regarding how content is used for training versus search results.
Cloudflare outlines these requirements as follows:
“It ‘must respect, via any mechanism, a ‘no training’ preference in robots.txt’
‘They give site owners a way to opt out of AI summaries’
‘They provide URL-level visibility into which pages were made available for training, as well metrics on search results, so you can see how your content was used for search and for training’
‘They can show publicly that Disallowing Training does not hurt your traditional search results'”
Failure to meet these criteria results in the crawler being treated as opaque and blocked across all sites using this setting. Notably, none of the major tech companies are named in the policy, though the conditions clearly target Google and Microsoft.
Evaluating Google’s Compliance
Condition four is already satisfied by Google, which has publicly stated that its Google-Extended crawler does not impact search rankings or inclusion. However, Condition two presents a more complex challenge. It requires a mechanism for site owners to opt out of AI summaries independently of standard search snippets.
Google’s current documentation indicates that controls like `nosnippet` or `max-snippet` apply globally to Search results, including AI Overviews. There is currently no setting that excludes a page from AI-generated answers while preserving its eligibility for standard search snippets. In contrast, Microsoft addressed this requirement in September 2023 by ensuring that content tagged `NOARCHIVE` is excluded from Bing Chat answers but remains visible in search results. This distinction highlights the varying levels of preparedness among major AI providers regarding publisher control.
Default Settings and Accountability
For new customers, Bot Preference Sync is enabled by default. As of mid-September 2026, Cloudflare updated the default configuration for new domains to block Training and Agent crawlers on pages displaying ads, while leaving Search allowed. This shift means that many new websites will automatically publish a stance against AI training without explicit manual intervention.
This raises accountability questions. Publishers who do not review their robots.txt files may find themselves enforcing policies they did not consciously choose. The author warns:
“The websites at risk are the ones that never open robots.txt again and never read the position written for them. They will have a policy on AI training that they did not write, cannot see, and could not have expressed in three settings anyway.”
Practical Steps for Website Owners
Given that robots.txt is a voluntary protocol respected only by well-behaved crawlers, it serves primarily as a declaration of intent rather than a hard security barrier. To mitigate risks associated with Bot Preference Sync, website owners should perform a quick audit:
- Compare Directives: Review your existing robots.txt file alongside your Cloudflare Security Settings. Identify any discrepancies between what the file says and what the dashboard enforces.While the tool offers convenience and helps maintain consistency for honest crawlers, it centralizes significant decision-making power within Cloudflare’s infrastructure. Publishers must remain vigilant to ensure their public crawl directives continue to serve their specific operational and legal needs.
