Summary of an article published on The Web Scraping Club on 2026-05-03 (free to read). Read the full article

Cloudflare has released its first API for crawling entire websites, the /crawl endpoint, currently in beta. It is pitched at RAG pipelines and website monitoring. The question I care about is whether it holds up in real-world web scraping, where sites sit behind WAFs, rate limits and anti-bot checks.

I went through how the endpoint works: the asynchronous job flow, the order in which it discovers URLs, the output formats and the pricing model, which is based on resource usage and not on pages. Then I went through the limitations, including how it treats bot protection, the User-Agent it sends, Content Signals and browser time caps on free plans. To show it in practice, I wrote a Python client with requests and ran it against the Quotes to Scrape sandbox, covering job creation, polling and paginated retrieval of the results.

What’s inside

  • How a crawl job is created, polled and read back, and how long job results stay available.
  • Why pricing is hard to predict: it follows the Browser Rendering or the Workers model depending on whether rendering is on.
  • Which crawl controls exist: depth and limit, include and exclude patterns, incremental fetching with cache-aware parameters, authentication and custom headers.
  • What happens when a site disallows AI use through Content Signals, or when the free plan’s daily browser time runs out.

In the full article: The full Python script with its configuration, the request parameters, the pricing tables, the terminal output of the run and the structure of the resulting records.json file.

Read the full article on The Web Scraping Club