Creative Commons Weighs ‘Pay-to-Crawl’ for AI Training
▼ Summary
– Creative Commons, known for its open licensing, is cautiously supporting “pay-to-crawl” systems to automate compensation for websites when AI bots scrape their content.
– The shift to AI chatbots has devastated publishers by reducing search traffic, as users get answers without clicking through to source websites.
– A pay-to-crawl system could help publishers, especially smaller ones, recover revenue and avoid restrictive paywalls, unlike one-off deals with major AI firms.
– Creative Commons outlined caveats, warning such systems could concentrate power and block access for public interest groups like researchers and educators.
– Several companies, including Cloudflare and Microsoft, are developing pay-to-crawl technologies, with a related standard called Really Simple Licensing (RSL) also gaining adoption.
The nonprofit organization Creative Commons, widely recognized for its open copyright licenses, is now exploring a “pay-to-crawl” model that could reshape how artificial intelligence companies access online content. This system would automate payments to websites whenever their material is scraped by AI bots for training purposes, offering a potential revenue stream for publishers who have seen traditional web traffic decline due to generative AI. While still in a conceptual phase, this approach aims to balance the needs of content creators with the data-hungry demands of the AI industry, seeking a sustainable path forward in a rapidly changing digital landscape.
Creative Commons has expressed cautious support for this emerging technology. The organization suggests that if implemented thoughtfully, such a system could help websites fund their operations and continue sharing content publicly, rather than retreating behind restrictive paywalls. The core idea, championed by firms like Cloudflare, involves charging AI bots a fee each time they extract content from a site to build or update machine learning models. This represents a significant shift from the longstanding practice where websites freely allowed search engine crawlers to index their pages, benefiting from the resulting traffic.
The dynamic has fundamentally changed with the rise of AI assistants. When a user receives an answer directly from a chatbot, they often have no reason to visit the original source website. This erosion of referral traffic has proven financially damaging for many publishers, with no indication the trend will reverse. A pay-to-crawl framework could help offset these losses. It might also level the playing field for smaller web publishers who lack the leverage to negotiate individual licensing deals with major AI firms, unlike large media conglomerates that have already struck agreements with companies like OpenAI, Axel Springer, and The New York Times.
However, Creative Commons attached several important caveats to its endorsement. It warned that pay-to-crawl systems could inadvertently concentrate power on the web and might block vital access for researchers, educational institutions, nonprofits, and others serving the public interest. To mitigate these risks, the organization proposed a set of responsible principles. These include ensuring pay-to-crawl is not a default setting for all sites, avoiding one-size-fits-all rules for the entire web, and designing systems that allow for throttling access rather than just outright blocking. The systems should also preserve public interest access, be built with open and interoperable standards, and use standardized components.
The push for new web-crawling standards is gaining broader industry momentum. Beyond Cloudflare’s efforts, Microsoft is developing an AI marketplace for publishers, and startups like ProRata.ai and TollBit are entering the space. Another initiative, led by the RSL Collective, has introduced a specification called Really Simple Licensing (RSL). This standard would allow websites to specify which parts of their content crawlers can access, without necessarily blocking them entirely. This approach has been adopted by major infrastructure providers like Cloudflare, Akamai, and Fastly, and is supported by companies including Yahoo and O’Reilly Media.
Creative Commons has also announced its support for the RSL standard. This aligns with its broader “CC signals” project, which focuses on developing the necessary technology and tools to navigate the challenges and opportunities presented by the AI era. The organization’s engagement signals a pivotal moment in defining how the open web interacts with the data requirements of advanced artificial intelligence, seeking solutions that compensate creators while maintaining essential public access to information.
(Source: TechCrunch)
