Summary of an article published on The Web Scraping Club on 2024-05-03 (for paid subscribers). Read the full article
“How can I bypass Cloudflare?” is the question I see most in every scraping community, and it comes without context. Cloudflare has the biggest market share among anti-bot solutions, and installations differ in strictness. A fix that works on one protected site may do nothing on the next. Before picking a tool, you need to know why your scraper is blocked.
I start with the diagnosis: run the same scraper from a residential connection and from a server, then separate browserless setups such as Scrapy with Scrapy-impersonate from browser automation with Selenium or Playwright, where fingerprinting changes the picture. Then I look at the open source side. I recall our earlier tests of Undetected Chromedriver against Cloudflare and other anti-bots, and I pick three free solutions to try, starting with Scrapy Impersonate, which I still run in production. Results always depend on the running environment and on each site’s rules.
What’s inside
- How to tell an IP reputation problem from a fingerprinting problem by changing where the scraper runs
- Why adding residential proxies is often enough for browserless scrapers but not for Selenium or Playwright
- What Undetected Chromedriver does and does not do, and why its maintainer limited the issue tracker
- Why no free tool works everywhere, given how much Cloudflare invests in R&D
In the full article: The three selected open source solutions with the reasoning behind each choice, and the code in the paid GitHub repository.