IAB Tech Lab Issues New Guidance on Managing AI Bots
IAB Tech Lab Steps Up to Address the AI Crawler Challenge
The rapid rise of artificial intelligence has brought with it an unexpected headache for publishers and content owners: a surge in AI-driven bots and crawlers scouring websites at an unprecedented scale. To help organizations navigate this evolving landscape, IAB Tech Lab has released new guidance on managing non-human web traffic, including automated agents and AI scraping systems. The document is open for public comment until June 26, 2026, giving industry stakeholders a chance to contribute before the framework is finalized.
The guidance arrives at a critical moment. Many publishers and digital businesses have yet to establish formal policies around bot and crawler management, even as concerns mount about AI systems straining server infrastructure, complicating advertising revenue, and consuming content without clear permission or compensation. IAB Tech Lab CEO Anthony Katsur emphasized that offering straightforward, actionable guidance is essential for meaningful industry adoption. The organization wants to help companies make informed decisions that align with their individual business goals while contributing to a healthier, more transparent digital marketplace. This initiative reflects a broader industry recognition that the questions surrounding AI access to web content are no longer theoretical — they are urgent, practical business decisions requiring clear frameworks and collaborative solutions.
Why Blanket Bot Blocking Is No Longer a Viable Strategy
One of the most significant takeaways from IAB Tech Lab’s new guidance is its position on blanket bot blocking. While it may seem like the simplest solution for publishers frustrated by unwanted crawlers, the organization argues that wholesale blocking is increasingly impractical. As AI Tools Integration becomes more deeply woven into the fabric of the web — from search engines to content recommendation platforms to automated research tools — indiscriminate blocking risks cutting off beneficial traffic alongside harmful scraping.
Instead, the guidance encourages a more nuanced approach. Publishers are advised to consider a spectrum of strategies, each carrying distinct operational, financial, and strategic implications. Some organizations may choose to permit certain AI crawlers under specific licensing or data-sharing agreements. Others might implement tiered access, allowing verified systems while restricting unidentified bots. The guidance is designed primarily for business leaders rather than engineers, recognizing that decisions about AI access often fall to executives who need clarity without deep technical expertise.
This balanced approach acknowledges that AI systems — including those that power Auto Backlinks Builder tools, content aggregators, and discovery platforms — are now integral parts of the digital ecosystem. Finding workable middle ground, rather than reactive blanket restrictions, is key to building a sustainable content economy for all parties involved.
CoMP API and the Future of AI-Publisher Communication
Alongside the new bot management guidance, IAB Tech Lab has been developing the CoMP API V1, a framework designed to facilitate structured communication and permissions between AI systems and content publishers. Together, these two initiatives represent a comprehensive effort to bring order to what has been a largely unregulated area of digital interaction.
The CoMP API essentially creates a standardized channel through which AI systems can identify themselves and request access to content under defined terms, while publishers can respond with clear conditions or restrictions. This kind of transparent infrastructure is vital as tools like AI Image Generator platforms, autonomous research agents, and large language model training pipelines increasingly rely on web-sourced data.
IAB Tech Lab’s EVP and COO Shailley Singh noted that content owners are being pushed to make consequential decisions rapidly, often lacking the frameworks needed to evaluate their options properly. The combination of practical bot management guidance and the CoMP API aims to fill that gap, empowering both sides of the equation — publishers protecting their assets and AI developers seeking legitimate data access.
The organization continues to invite input from members and industry participants as it refines both initiatives, underscoring its commitment to building consensus-driven standards that reflect the real-world complexity of today’s AI-integrated digital environment.
Source: IAB Tech Lab tackles the growing AI bot problem | MarTech
