Cloudflare launches a market that allows websites to load robots ai to scratch

Cloudflare, a cloud infrastructure provider that serves 20% of the web, announced on Tuesday the launch of a new market that reinvents the relationship between websites and AI companies – ideally giving publishers a greater control over their content.

In the past year, Cloudflare has launched tools for publishers to fight Solution with one click to block all the robots AIas well as Dashboard to see how the crawlers have visited their site. In an interview in 2024, Cloudflare CEO Matthew Prince told Techcrunch that these products threw a base for a new type of market in which publishers could distribute their content to AI companies and be compensated.

Now Cloudflare gives life to this market.

This is called Pay Per Crawl, and Cloudflare launches on Tuesday “the experience” in private beta version. Website owners in the experience can choose to leave AI tracks, on an individual basis, to scratch their site at a defined pace – a micropament for each “ramp”. Alternatively, websites can choose to leave the Crawlers operating robots for free to scratch their site or block them completely. Cloudflare claims that its tools will allow sites of sites to see if robot robots are erafonent their site for AI training data, to appear in AI search responses or for other purposes.

On a scale, the Cloudflare market is a big idea that could offer publishers a potential business model for the AI ​​era – and it also places Cloudflare in the center of all of this. The launch of the market comes at a time when news publishers are faced with existential questions about how to reach readers, while Google research traffic is fading and AI chatbots increase in popularity.

There is no clear answer on how news publishers will survive the AI ​​era. Some, like the New York Times, have deposited Prosecution against technological companies For the training of their AI models on press articles without authorization. In the meantime, other publishers have concludes multi -year offers to concede to their content For the formation of the AI ​​model and so that their content appears in the AI ​​chatbot responses.

Despite this, only the major publishers have concluded license agreements on AI, and it is not yet clear if they provide significant sources of income. Cloudflare aims to create a more sustainable system where publishers can set prices to their own conditions.

The company also announced Tuesday that the new websites set up with Cloudflare would now block all the Robots of the AI. The owners of sites will have to grant certain crawlers the permission of access to their site – a so -called cloudflare change will give each new domain “the lack of control”.

Several large publishers, including Conde Nast, Time, the Associated Press, the Atlantic, Adweek and Fortune, signed with Cloudflare to block nursing robots by default in support of the company’s wider objective of a “approach based on permission to crawl”.

The business model on which many of these publishers have relied for decades become slowly unreliable. Historically, online publishers allowed Google to scratch their sites in exchange for references in Google Search, which has resulted in traffic to their sites, and ultimately, advertising revenues.

However, Cloudflare’s new data suggest that publishers can get a worse affair in the AI ​​era than in the Google research era. While Some websites cite chatgpt as a major traffic sourceThis does not seem to be the case widely.

In June, Cloudflare said he found that the Google robot had scratched his websites 14 times for each reference he had given them. Meanwhile, the Openai opening robot has scratched the websites 17,000 times for each reference, while Anthropic has scratched the websites 73,000 times for each reference.

Meanwhile, Openai and Google build AI agents who are designed for Visit websites on behalf of usersCollect information and give it directly to users. A future in which these tools are dominant have enormous implications for publishers who rely on readers visiting their sites.

Cloudflare notes that the “real potential” of ramp remuneration can emerge in the “agentic” future.

“What happens if an agent payment wall could work at the forefront of the network, entirely by program? Imagine asking your favorite depth research program to help you synthesize the latest research on cancer or a legal memory, or simply help you find the best Soho restaurant-then give this agent a budget to spend to acquire the best and most relevant content,” said Cloudflare in a blog article.

To participate in the Cloudflare experimental market, AI companies and publishers must both be configured with Cloudflare accounts. In their accounts, the two parties can set prices to which they would like to buy and sell a “ramp” of the editor’s content. Cloudflare acts as an intermediary in these transactions, invoicing the company of AI and distributing profits to the publisher.

Cloudflare spokesperson Ripley Park told Techcrunch that there are no stabaces or cryptocurrency involved in the salary by ramp at the moment, even if many have suggested Digital currency would be perfect for something like that.

The Cloudflare market looks like a bold vision of the future which requires that many editors and AI companies get on board. However, there is no guarantee that publishers will get a good deal, and convince IA companies to participate could be difficult, since they currently scratch the content for free.

Nevertheless, Cloudflare seems to be one of the few companies able to ensure that a market like this occurs.