Summary of an article published on The Web Scraping Club on 2023-10-12 (for paid subscribers). Read the full article
Cloudflare holds more than 86% of the anti-bot market according to Statista, so on a medium-large project you will meet it. A plain Scrapy spider gets a 429 and stops. I had moved from headful Playwright to web unblockers, because a browser costs more resources and time and the bill ended up close anyway. Then, as always in this cat-and-mouse game, the setup broke: sites protected by Cloudflare and Datadome stopped working through the unblockers.
In this episode of The Lab I explain how I think Cloudflare decides who gets challenged. I don’t reverse-engineer its API calls. My view comes from trial and error and the general principles of bot detection. I cover Turnstile and the signals behind it: IP reputation and type, fingerprinting, and JavaScript challenges. Then I move to the Scrapy Impersonate package, which I found while working on the previous post, to get Scrapy past it.
What’s inside
- How to check whether a target uses Cloudflare with the Wappalyzer Chrome extension.
- Why Turnstile relies on detection and not on a task to complete like a proof-of-work Captcha, and what that means for scrapers and for users behind a VPN or proxy.
- Why Playwright on a data center IP needed residential proxies on some sites, and how that changed the cost comparison with the cheapest unblockers.
- A caveat on the method: every website sets its own Cloudflare rules, so a solution that works on one target can fail on another.
In the full article: The Scrapy Impersonate spider code and its implementation, kept in The Lab’s GitHub repository.