Cloudflare is now blocking AI robots by default

The main provider of Internet Cloudflare will now block known web robots by default to prevent them from “accessing content without authorization or compensation”, according to an ad Tuesday. With the change, Cloudflare will begin to ask new domain owners if they want to authorize the AI ​​grayfectors, and will even allow certain publishers to implement “salary by handle”.

The Pay Per Crawl program will allow publishers to set a price for AI Grabyers to access their content. AI companies can then view prices and choose to register for “Pay Per Crawl” costs or turn away. This is only available for “a group of some of the main editors and content creators” for the moment, but Cloudflare says that it will guarantee that “AI companies can use quality content in the right way – with permission and compensation”.

Cloudflare has been helping domain owners to fight AI robots for some time now. The company began to leave websites Block AI robots in 2023, but he only applied themselves to those who respect Robots.txt file of a siteThe inapplicable agreement that points out whether robots can scrape its content. Cloudflare started Allow websites to block Last year, “All”, whether or not they respect the Robots.txt file of a site – and now this parameter is activated by default for new Clients Cloudflare. (The company identifies the scrapers to be blocked by comparing them to its list of known robots.) Cloudflare also deployed a functionality in March Who sends web bush bots In an “AI Labyrinth” to dissuade them from scratching the sites without authorization.

Several major publishers and platforms, including The Associated Press,, The Atlantic,, FortuneStack Overflow and Quora are on board with the new Cloudflare IA Crawler restrictions, because websites face a future where more people find information via IA chatbots, rather than search engines. “People trust AI more in the last six months, which means that they do not read the original content”, the CEO of Cloudflare Matthew Prince said during the Axios event live last week.

In addition, Cloudflare says that it works with AI companies to help check their robots and allow them to “clearly state their objective”, for example if they use content for training, inference or research. Website owners can then consult this information and determine the robots to let in.

“The original content is what makes the Internet one of the greatest inventions of the last century, and we have to come together to protect it,” Prince said in the press release. “AI crawlers have scratched limitless content. Our goal is to put power in the hands of creators, while helping IA companies to innovate. ”