Summary of an article published on The Web Scraping Club on 2023-04-14 (for paid subscribers). Read the full article

Datadome is gaining popularity as an anti-bot solution, and it shows up on large sites you may need in a project. It combines device fingerprinting, behavior analysis and machine learning, so a basic Scrapy spider with no JavaScript rendering has no chance. Bypassing it has a cost, especially when several big websites are involved. This is the third post in our anti-bot series, after Kasada and Cloudflare.

We start with how to recognize Datadome on a target, then test free open-source tools against Footlocker.it, aiming at the men’s shoes product list page. We ran Playwright with Chrome, Firefox and Brave, both on a local machine and on a VM in a datacenter, and tried a package that simulates human mouse movements. We close with the cost trade-off between headful browsers and commercial APIs called from a simple Scrapy spider.

What’s inside

  • Three ways to spot Datadome: Wappalyzer, the cookie set on the first response, and the slider challenge you can get in Incognito mode.
  • python_ghost_cursor simulates human mouse movements with Bezier curves, and it only works with Chrome.
  • Each browser setup was run from both a local environment and a datacenter VM, since results vary with target configuration and environment.
  • Headful browsers need more CPU and memory, while commercial APIs charge per call but cut the cost of the machines hosting the scrapers.

In the full article: The scraper code and screenshots for each Playwright setup, and the result per browser and environment on Footlocker.it.

Read the full article on The Web Scraping Club