AI’s Free Web Scraping Era Ends With New Licensing Protocol
▼ Summary
– Several major publishers and tech companies have developed the Really Simple Licensing (RSL) standard to address AI companies extracting web content without permission or compensation.
– RSL allows publishers to define machine-readable licensing terms, specifying requirements like attribution, pay-per-crawl, or pay-per-inference for AI use of their content.
– The RSL Collective, modeled after music rights organizations, will negotiate with AI companies on behalf of publishers and creators to ensure fair compensation.
– RSL is an open protocol that any web publisher can use, from large outlets to individual creators, to specify their licensing terms for AI crawlers.
– The protocol aims to evolve web standards by providing a licensing framework that gives content creators more control and bargaining power in the AI economy.
A new licensing protocol is poised to fundamentally reshape how artificial intelligence companies access and use online content, potentially ending the era of unrestricted web scraping. Major publishers and tech firms including Reddit, Yahoo, People, O’Reilly Media, Medium, and Ziff Davis have united to introduce the Really Simple Licensing (RSL) standard, a framework designed to give content creators control and compensation when their work is used by AI systems.
Think of RSL as RSS’s more assertive counterpart. Where RSS facilitated the free flow of content across the web, RSL establishes clear, machine-readable rules for AI crawlers. Publishers can now specify whether their content is free to use, requires attribution, or must be paid for, whether per crawl or even per AI-generated inference. This marks a dramatic shift from the simplistic and often ignored robots.txt approach, replacing vague permissions with enforceable, granular licensing terms.
The protocol introduces a shared vocabulary for licensing and compensation, enabling everything from free use with attribution to pay-per-crawl or pay-per-inference models. It also supports open automation for content licensing, public catalogs of licensable material, and encryption for proprietary or paywalled assets. Perhaps most significantly, RSL enables collective licensing through organizations like the RSL Collective, which aims to negotiate on behalf of publishers and creators, much like ASCAP or BMI do in the music industry.
Tim O’Reilly, a key advocate for the initiative, notes that while RSS was essential in scaling the early internet, the rise of AI has created a new dynamic where content is ingested and repurposed without permission or payment. The current model, he suggests, is unsustainable. As AI companies train models on vast quantities of web content without compensating creators, publishers find themselves at a severe disadvantage. A collective approach could rebalance the scales.
The business implications are substantial. Individual creators and smaller outlets have little leverage against tech giants, but a unified front representing millions of content pieces could negotiate favorable terms. The RSL Collective offers a centralized mechanism for rights management, allowing even independent publishers to participate in revenue sharing and licensing agreements.
This development arrives amid growing legal and ethical scrutiny over how AI models are trained. For years, AI developers have harvested online data largely without restriction, relying on an advertising-driven web economy that has since eroded. Now, as generative AI attracts massive investment while publishers struggle, RSL proposes a structural solution: embedding licensing directly into the web’s infrastructure.
Guided by a technical committee including architects behind RSS and Schema.org, RSL aims to become a foundational web standard, akin to HTTP or HTML, but with a focus on rights and remuneration. If adopted widely, it could ensure that human creators remain relevant and rewarded in an increasingly automated digital landscape.
This isn’t just a technical update; it’s a cultural and economic correction. By making licensing machine-readable and universally applicable, RSL could help preserve the ecosystem that fuels AI, ensuring that the feast of online content doesn’t leave the kitchen empty.
(Source: ZDNET)
