Cloudflare sets September deadline for AI crawlers to pay

Cloudflare is drawing a line in the sand for the artificial intelligence industry. Starting in September, the company will automatically block the AI crawlers that scrape web content for training purposes. Any page that displays advertising will be off-limits, unless the site’s owner explicitly grants permission. The message is blunt: the free ride for harvesting the web is over.

As a gatekeeper for a massive portion of global internet traffic, Cloudflare announced the policy shift on Wednesday. Beginning 15 September, any new site built on Cloudflare will continue to allow search engines to index its pages. However, by default, those same sites will block AI training bots and AI agent crawlers from accessing any page that runs ads.

The new rules also target so-called “mixed-use” crawlers, which bundle search indexing, AI training, and agent tasks into a single bot. If a crawler refuses to let site owners separate these functions, it will be blocked on ad-supported pages. These defaults apply to all new customers, new sites from existing clients, and every free-tier user who has not manually adjusted their settings. Site owners retain the ability to re-enable bots from their dashboard, but the default posture has been reversed. Content that generates revenue is now locked down for AI unless the owner chooses to unlock it.

Why the urgency now

Cloudflare’s reasoning is rooted in a startling statistic. Automated bots now account for more than half of all web traffic, a threshold the company says arrived sooner than anticipated. CEO Matthew Prince noted that most internet traffic is no longer human. Cloudflare, he argued, “must go further and act faster so that a sustainable ecosystem can emerge.”

The core dilemma is one publishers know intimately. Most sites want visibility in AI-generated answers, just as they crave high rankings in traditional search. Yet the same crawl that indexes a page often feeds a model that then answers the user directly. The user never visits the site, and the ad revenue never materializes.

Cloudflare took a pointed swipe at the “world’s largest search engine,” clearly referring to Google. Its Googlebot combines indexing with AI training, giving Google roughly double the data access of rival AI firms. Blocking the bot risks disappearing from search results entirely. Microsoft’s Bing and Apple’s Applebot present the same painful trade-off.

From a tollbooth to a meter

Blocking crawlers is only half the strategy. Cloudflare is evolving last year’s “Pay Per Crawl” model into a “Pay Per Use” system. It now compensates publishers when their content directly shapes an AI-generated answer, not just when it is fetched for training. Early partners include the AI search firms Ceramic.ai and You.com.

Cloudflare is also rolling out a dashboard that lets publishers see exactly which bots are taking their content and how little traffic those firms send back. The company has coined a term for this behavior: Answer Engine Optimisation (AEO) , positioning it as the AI-era successor to SEO.

This reframing arrives in a market already leaning in the same direction. A wave of startups now sells tools to help brands maintain visibility inside chatbots, betting that GEO is the next SEO. Cloudflare aims to be the infrastructure layer beneath it.

The open web hangs in the balance

The backdrop for publishers is bleak. AI-generated answers are steadily eroding the clicks that fund the open web. They keep users inside Google’s ecosystem or within a chatbot, rather than sending them to the original content creators. One field study found that Google’s AI Overviews reduced outbound clicks by roughly 40 per cent. Economists have begun modeling scenarios in which the open web collapses entirely if the current bargain is not repaired.

Whether one company can fix this is uncertain. Google and Apple already offer opt-out crawlers that may bypass Cloudflare’s block, and competitors could find workarounds. Regulators are circling the same problem from a different angle. The UK is pushing Google to let publishers opt out of AI search without losing their ranking, and news publishers are suing OpenAI over training data.

Cloudflare’s move is the most aggressive attempt yet to force AI companies to pay for the content they consume. The deadline is 15 September. The rest of the web will be watching closely to see how the AI giants respond.