Summary of an article published on The Web Scraping Club on 2024-03-28 (for paid subscribers). Read the full article
Browser fingerprinting is the one challenge I usually hand to commercial tools, like unblockers or anti-detect browsers. A scraper running from a datacenter leaks plenty of signals: a WebRTC address, a timezone that doesn’t match the IP, a software renderer typical of servers without a GPU. Proxies alone don’t fix any of that. I wanted to know how much of the job an open source Python tool can do before paying for a commercial one.
We tested Browserforge, a Python reimplementation of Apify’s fingerprint suite, together with Playwright. From an AWS server in Berlin, with residential proxies, we loaded a BrowserScan check and then pages on Harrods (Cloudflare), Footlocker (Datadome) and StockX (PerimeterX). We ran several configurations, from default Chromium to injected fingerprints in different browsers, and compared them with a Kameleo profile.
What’s inside
- Why I follow a Buy AND Make approach to anti-bot challenges, and why I leave browser fingerprinting to commercial tools when a site relies on it heavily.
- What happens when the browser engine and the injected fingerprint don’t match, for example Chromium carrying a Firefox on Windows fingerprint.
- How the system timezone and the proxy country have to line up with the fingerprint, and what we changed on the Berlin server.
- Where Browserforge falls short of an anti-detect browser: the browser version can’t be chosen, and WebRTC can’t be switched off programmatically in traditional browsers.
In the full article: Code for every run from The Lab repository (folder 46), the BrowserScan screenshots for each configuration, the Kameleo comparison and the results against Cloudflare, Datadome and PerimeterX.