Summary of an article published on The Web Scraping Club on 2025-08-24 (free to read). Read the full article
Cloudflare introduced Pay per Crawl and began blocking AI crawlers by default. The buzz covers little of how AI companies actually scrape. Their budgets are unlike anything we saw before, and site owners pay for the infrastructure that absorbs the load. LLM answers also send fewer people to the original page, even for sites ranking first on Google. Paying per crawl sounds good in theory.
I ran the idea through a simple model: a small bike repair guide site, a per-request price and a few AI companies crawling it. Then I questioned two assumptions behind it, using Google’s index size to size the scraping volume and Bright Data’s Web Unlocker as the alternative AI companies already have. I also cover Databoutique.com as a different model, and bot detection through Cloudflare’s report on Perplexity and a LinkedIn post by Antoine Vastel.
What’s inside
- Garry Tan’s report that Cloudflare blocked AI crawlers on the YCombinator domain without him noticing.
- The costs on the owner’s side beyond the fees: running Cloudflare itself and the cut taken by the payment platform used in the beta.
- The tension between a price attractive to AI companies and a price worth collecting for the site owner.
- Why whitelisting LLM user agents without checking IPs is a weak setup, given how easy it is to change a user agent.
In the full article: The full revenue calculation for the bike repair site, the estimate of yearly AI request volume behind it, the price comparison with the Bright Data unblocker and the Databoutique model in more detail.