Summary of an article published on The Web Scraping Club on 2025-01-23 (for paid subscribers). Read the full article

Sooner or later every scraper meets Cloudflare, and Statista gives it about 80% of the bot protection market. My gut feeling is that the number is overestimated, but the problem stays the same: we need a way past it. On the Burger King reviews page on Indeed.com, Turnstile runs a back-and-forth of POST requests with encrypted payloads, most likely our browser fingerprint. Then it hands out a clearance cookie, valid for six months in this case.

I explain why reverse engineering that process, as Cloudscraper tried, teaches a lot but is a poor way to run a real project. Then I cover the alternative of behaving like a real browser, with genuine fingerprints in Playwright or Puppeteer, which is what anti-detect browsers do. Finally we test open-source tools on the same Indeed page: Playwright, Cloudscraper and Botasaurus. The same test with commercial tools comes in the following months.

What’s inside

  • Why reverse engineering is an unfair game: every anti-bot needs its own process, and each update sends us back almost to the start.
  • Where fingerprint matching stops being enough: strict anti-bot rules that add keyboard and mouse behavior tracking.
  • The setup of the Playwright test: a local Brave instance with masking arguments and no stealth library added.
  • The state of Cloudscraper, whose repository has not been updated for two years.

In the full article: Scripts for each tool, the result of each test on the Indeed page, the Botasaurus section and the 73.CLOUDFLARE folder of the GitHub repository.

Read the full article on The Web Scraping Club