Topic: mixed-use crawlers
-
Cloudflare’s new policy requires AI firms to pay publishers for content
Cloudflare will automatically block "mixed-use" AI crawlers that combine search indexing, AI agent use, and model training on ad-supported pages starting September 15, 2026, unless site owners manually adjust settings. The policy targets blended crawlers like Google's, which Cloudflare claims has...
Read More » -
Cloudflare sets September deadline for AI crawlers to pay
Starting September 15, Cloudflare will automatically block AI crawlers from accessing ad-supported web pages by default, reversing the prior open-access posture unless site owners explicitly grant permission. Automated bots now account for over half of web traffic, and Cloudflare argues AI compan...
Read More » -
Patreon drops request, starts blocking AI scrapers
Patreon has partnered with Cloudflare to actively block AI training bots from scraping creators' content, moving beyond voluntary robots.txt restrictions to mandatory controls. The crackdown responds to increasingly aggressive scraping tactics and new discovery features like Home Feed and Quips t...
Read More »