Summary of an article published on The Web Scraping Club on 2026-05-03 (free to read). Read the full article
Cloudflare has released its first API for crawling entire websites, the /crawl endpoint, currently in beta. It is pitched at RAG pipelines and website monitoring. The question I care about is whether it holds up in real-world web scraping, where sites sit behind WAFs, rate limits and anti-bot checks.
I went through how the endpoint works: the asynchronous job flow, the order in which it discovers URLs, the output formats and the pricing model, which is based on resource usage and not on pages. Then I went through the limitations, including how it treats bot protection, the User-Agent it sends, Content Signals and browser time caps on free plans. To show it in practice, I wrote a Python client with requests and ran it against the Quotes to Scrape sandbox, covering job creation, polling and paginated retrieval of the results.
What’s inside
- How a crawl job is created, polled and read back, and how long job results stay available.
- Why pricing is hard to predict: it follows the Browser Rendering or the Workers model depending on whether rendering is on.
- Which crawl controls exist: depth and limit, include and exclude patterns, incremental fetching with cache-aware parameters, authentication and custom headers.
- What happens when a site disallows AI use through Content Signals, or when the free plan’s daily browser time runs out.
In the full article: The full Python script with its configuration, the request parameters, the pricing tables, the terminal output of the run and the structure of the resulting records.json file.