IAB Tech Lab takes on rising AI bot issue

The explosion of AI crawlers and automated agents has created a new dilemma for publishers and content owners: determining who gets access to their material and under what conditions. This challenge, which touches on everything from server costs to revenue models, now has a formal response from the industry.

IAB Tech Lab has released fresh guidance on bot and crawler management, opening the document for public comment until June 26, 2026. The framework aims to help publishers, content owners, and ad tech companies navigate the complexities of non-human traffic, including AI systems that scrape or access content. It builds on the organization’s recently launched CoMP API V1, a standard designed to facilitate communication and permissions between AI systems and publishers.

“Giving content owners clear, practical guidance is key if we want adoption to move forward in a meaningful way,” said Anthony Katsur, CEO of IAB Tech Lab, in a statement. “This work helps simplify a complex area so companies can make decisions that fit their business while supporting a more sustainable marketplace.”

The paper underscores how rapidly AI crawling has become a business-critical issue. Many organizations still lack formal policies on bots and crawlers, even as concerns mount about AI systems consuming content, straining infrastructure, and complicating monetization. According to IAB Tech Lab, the guidance was created because publishers and content owners were struggling to weigh the trade-offs of managing AI access. Blanket bot blocking, the group argues, is no longer practical as AI systems become more embedded across the web ecosystem.

Instead, the document outlines a spectrum of approaches, each with its own operational, financial, and strategic implications. The guidance is aimed less at engineers and more at business leaders who need to understand the costs and consequences of AI crawler management.

“Content owners are being asked to make important decisions quickly, often without clear frameworks,” said Shailley Singh, EVP and COO at IAB Tech Lab. “This guidance helps break down the options so they can choose an approach that aligns with their goals.”

IAB Tech Lab plans to continue working with members and industry participants to support implementation of both the bot management guidance and the CoMP API. The organization is accepting public comments on the proposal through June 26 before finalizing the framework.