Dopious
Senior Member
Founding Member
Sapphire Member
Patron
Bronze Star
Bronze Star
Bronze Star
Bronze Star
Bronze Star
Cloudflare is now automatically blocking AI crawlers for new customers and is launching “Pay Per Crawl,” where you as a site owner decide whether you want to block, charge, or provide free access to AI crawlers. You set the price yourself, there are no recommended levels, minimum or maximum amounts, and it is unclear whether Cloudflare charges any fees. Exactly how it works is not yet public, as the service is in private beta and only open for applications.
Publishers that have already signed up to Pay Per Crawl include major brands such as TIME, BuzzFeed, Quora, Stack Overflow, The Atlantic, and Webflow. These players will now have the ability to protect their material and create new revenue streams while controlling how their content is used by AI services.
Cloudflare CEO Matthew Prince justifies the change by saying that it has become significantly more difficult to be a content creator on the web: “Ten years ago, for every two pages Google fetched, you got one visitor. Today, for every 18 pages Google takes away from you, you get one visitor.” (Translated quote from the interview with Axios, see the video below). He points out that developments are rapidly moving in the wrong direction, especially with AI-powered search results that often provide the answer directly, thereby reducing traffic to the original source.
For smaller players, it can be a challenge to get paid or appear in AI-powered search results if they set high prices, but it is still possible to allow free access and maintain visibility. For companies that do not use Cloudflare, this does not affect them at all.
Technically, the solution is based on the HTTP status code 402 Payment Required. When an AI crawler tries to retrieve content, it is either granted access directly, or is met with a 402 page with information about the cost. The crawler can then choose to accept the price and pay, or forgo it. All payment handling and logging of crawl events is handled by Cloudflare, which acts as the Merchant of Record.
The Pay Per Crawl service is still in private beta and you can apply for access here.
Now go and make some cash with your high traffic sites.
Publishers that have already signed up to Pay Per Crawl include major brands such as TIME, BuzzFeed, Quora, Stack Overflow, The Atlantic, and Webflow. These players will now have the ability to protect their material and create new revenue streams while controlling how their content is used by AI services.
Cloudflare CEO Matthew Prince justifies the change by saying that it has become significantly more difficult to be a content creator on the web: “Ten years ago, for every two pages Google fetched, you got one visitor. Today, for every 18 pages Google takes away from you, you get one visitor.” (Translated quote from the interview with Axios, see the video below). He points out that developments are rapidly moving in the wrong direction, especially with AI-powered search results that often provide the answer directly, thereby reducing traffic to the original source.
For smaller players, it can be a challenge to get paid or appear in AI-powered search results if they set high prices, but it is still possible to allow free access and maintain visibility. For companies that do not use Cloudflare, this does not affect them at all.
Technically, the solution is based on the HTTP status code 402 Payment Required. When an AI crawler tries to retrieve content, it is either granted access directly, or is met with a 402 page with information about the cost. The crawler can then choose to accept the price and pay, or forgo it. All payment handling and logging of crawl events is handled by Cloudflare, which acts as the Merchant of Record.
The Pay Per Crawl service is still in private beta and you can apply for access here.
Now go and make some cash with your high traffic sites.
