Cloudflare launched Pay per Crawl: AI bots will pay for scanning sites
Cloudflare company launched a very interesting service! It allows website owners to charge AI bots money for data collection! The service is called Pay per Crawl and is currently running in closed beta version.
It’s important to understand the scale. Cloudflare serves 20% of the entire internet. And through this platform, website owners will be able to choose. Whether to allow AI bots to scan their content for free, block them completely, or charge micropayments for each scan.
The service makes it clear what exactly bots are collecting information for. For training AI models, for appearing in chatbot responses, or for other purposes.
Major publications have already joined the initiative – Conde Nast, TIME, Associated Press and others. They will block AI bots by default.
Statistics from Cloudflare show a curious picture. Google search engine scans sites 14 times for each received transition. And OpenAI’s bot – 17,000 times for 1 transition, and Anthropic – 73,000 times!
