Cloudflare Launches AI Bot Blocker to Help Publishers Get Paid

Cloudflare, which powers around 20% of the internet, is taking a stand against unauthorised AI bots by blocking them from scraping websites, in a move aimed at protecting content creators and media publishers from being exploited by AI firms.

Cloudflare is blocking AI crawlers like OpenAI’s GPTBot and Anthropic’s ClaudeBot by default unless website owners explicitly allow access.

The company is also piloting a “Pay Per Crawl” model, enabling site owners to charge AI companies that want to train models on their content.

AI bots have been accused of extensively crawling the web, often ignoring established protocols and extracting vast amounts of content without consent or payment.

The resulting AI-generated summaries, like Google’s AI Overviews, have led to a drop in referral traffic and ad revenue for content-heavy sites.

Major publishers including Condé Nast, Dotdash Meredith, and Reddit have shown their support for Cloudflare’s initiative.

“It sets a new standard for how content is respected online,” said Condé Nast CEO Roger Lynch. “Permission and partnership must be at the heart of sustainable innovation.”

The move comes as the debate over AI’s use of copyrighted content intensifies globally. While some publishers have struck licensing deals with AI firms, others, including The New York Times, have turned to the courts.

Cloudflare’s default block, along with tools to detect “shadow” crawlers, marks a potential turning point in the AI-content battle.

AI vendors will now need to ask and pay before taking people’s data, content, or work to train their models.

“This is about giving publishers control and building a new internet economy that rewards content creators,” said Cloudflare CEO Matthew Prince. “Original content powers the web, and creators deserve compensation.”