Cloudflare is introducing more nuanced controls for managing AI bots on websites, replacing its previous blanket block feature with three categories: search, training, and agent crawlers. This change allows site owners to fine-tune how AI bots access their sites, with Training and Agent bots being blocked by default on ad-carrying pages. The move aims to push bot operators to be more transparent about their purposes, and it’s a significant development in the ongoing evolution of internet traffic, which recently saw bot traffic surpass human traffic for the first time, why it matters is that this shift reflects the growing need for more sophisticated management of AI-driven web interactions.
Cloudflare Refines AI Bot Controls
Cloudflare is introducing more nuanced controls for managing AI bots on websites, replacing its previous blanket block feature with three categories: search, training, and agent crawlers. This change allows site owners t…
Summarized from
The Decoder AI-summarized
Was this helpful?
Discussion
Join the conversation
Comments are powered by GitHub Discussions. You need a GitHub account to participate.
The sharpest AI news, curated daily.
One email each morning. Summarized, sourced, and skimmable in under three minutes.
The sharpest AI news, curated daily.
One email each morning. Summarized, sourced, and skimmable in under three minutes.