Summary of an article published on The Web Scraping Club on 2023-01-19 (for paid subscribers). Read the full article

Cloudflare Bot Management is one of the strongest and most used anti-bot systems. It relies on machine learning over request patterns, user agents, IPs and fingerprints, it is updated constantly, and it is customized per site, so there is no silver bullet. In my previous Cloudflare post I wrote three similar solutions for three websites, and only two still work. On Antonioli, Playwright got blocked after a few pages, especially from a VM on AWS, even after trying different browsers, contexts and cloud providers.

We then tried Playwright launching an anti-detect browser. I describe how these Chromium forks differ from plain Chrome, the three requirements I set to choose one, and why GoLogin fit. Then I go through the trial setup with a Windows workstation profile, and a fingerprint comparison on amiunique.org between standard Playwright on my Mac and the GoLogin profile.

What’s inside

  • Why Cloudflare’s behavioral detection is hard to beat: request bursts, user agent and IP patterns, inconsistent fingerprints, and constant model updates.
  • What the server sees with Chrome under Playwright (a genuine Chrome on a datacenter machine) versus a custom profile with its own device fingerprint.
  • The three selection criteria: a fully working free demo, quick Playwright integration, and a Unix client for production.
  • Why being able to create multiple profiles, each with a different fingerprint, mattered for the experiments.

In the full article: The Playwright and GoLogin code from The Web Scraping Club GitHub repository, plus the full amiunique.org screenshots and the other fingerprint differences between the two setups.

Read the full article on The Web Scraping Club