Cloudflare’s new policy requires AI firms to pay publishers for content

Cloudflare is drawing a firm line in the sand for the AI industry. The company announced on Wednesday that, effective September 15, 2026, its default settings will automatically block any crawler that mixes traditional search indexing with AI agent use and model training on pages that display ads. This policy shift targets what Cloudflare calls “mixed-use” crawlers, forcing AI firms to separate their bots for search, agent services, and training into distinct, transparent categories.

Under the new defaults, these blended crawlers will be blocked from accessing ad-supported pages unless a site owner manually adjusts the settings. The change applies to new Cloudflare customers, new sites from existing customers, and all existing free-tier users. For AI companies, this could significantly alter how they gather web content for training and powering agentic services.

Cloudflare acknowledges that most website owners want their content discoverable through search and even AI tools, but they also want safeguards against their intellectual property being used without compensation. The company specifically called out the “world’s largest search engine,” a clear reference to Google, noting that it has access to “about 2x more information” than other AI firms because it makes it difficult for customers to remain discoverable without also being used for AI training.

Google has previously pushed back, pointing to its Google Extended bot, which lets site owners opt out of having content used for training and AI products like Gemini Apps and Vertex API. That opt-out does not affect a site’s inclusion in Google Search. However, Google’s flagship Googlebot still crawls for Search, including AI features such as AI Overviews and AI Mode.

“Now that the majority of traffic on the Internet is non-human, we must go further and act faster so that a sustainable ecosystem can emerge,” said Cloudflare co-founder and CEO Matthew Prince in the announcement, referencing the recent milestone where bots surpassed human traffic online for the first time. That shift was not expected to occur until next year.

Prince added that Cloudflare’s new tools and partnerships give website owners “increased visibility and commercial opportunities” while benefiting AI companies that operate with “clear and transparent intent.” He expressed hope that the default changes will push mixed-use crawlers to separate search from agent use and training.

Beyond blocking, Cloudflare is also expanding its commercial tools for publishers. The company previously launched a marketplace called Pay Per Crawl, allowing websites to charge AI bots for scraping. That concept is now evolving into Pay Per Use, which will let publishers charge AI companies when their content actually creates value, not just when it’s fetched.

Cloudflare’s data suggests that over 50% of crawl traffic from AI crawlers is wasted on re-fetching unchanged pages, so the change could also help conserve bandwidth and compute resources for AI model providers.

To put the new model into practice, Cloudflare is initially working with two partners: Ceramic.ai and You.com. When a publisher opts in, they are paid when their content appears in Ceramic’s AI search results or when You.com accesses a piece of their premium content. Cloudflare says other AI companies can customize this model for how they operate.