Summary of an article published on The Web Scraping Club on 2024-04-16 (free to read). Read the full article

Fingerprinting, rate limiting, geolocation, WAFs, challenges and CAPTCHAs all exist to stop automated traffic. Websites build a digital fingerprint from device, browser and connection data, and the protection system decides from it whether you look suspicious. Changing one parameter, like the timezone or the screen resolution, does not make you a new user. That is the practical problem when you scrape at scale.

With our partner Octo Browser, we look at where an anti-detect browser fits in a scraping stack. I explain what a fingerprint contains, including Canvas and WebGL hashing, and how profiles isolate cookies, proxies and fingerprint parameters. We compare spoofing by injected JavaScript with spoofing at the browser kernel level. We cover automation through the Chrome DevTools protocol with Puppeteer, Playwright and Selenium, show an example on hyatt.com, protected by Kasada, and discuss cost. We also list what else you need next to the browser: proxies, human-like behavior and CAPTCHA solvers.

What’s inside

  • Three levels of masking for large-scale scraping: IP address, digital fingerprint, and bot behavior.
  • A checklist for choosing a multi-accounting browser, including spoofing quality, kernel update frequency, API capabilities, profile costs and support.
  • Why a Linux distribution on a server can beat a home host for this kind of scraping.
  • Where costs go up (profile licenses) and where they go down (fewer blocks, less manual work), and what to look for in proxies, such as spam scores and residential latency.

In the full article: The list of fingerprint checkers, the before and after screenshots of the hyatt.com test, the links to the Octo Browser documentation and API instructions, and the full set of selection criteria for the browser, proxies and CAPTCHA solvers.

Read the full article on The Web Scraping Club