Summary of an article published on The Web Scraping Club on 2023-08-17 (for paid subscribers). Read the full article
PerimeterX, now part of Human, is one of the most widespread anti-bot solutions on the web. It combines fingerprinting on several layers, IP filtering and AI applied to behavioral analysis, so a scraper that only fixes one of these layers usually still gets blocked. Neimanmarcus.com is a good example of a target protected this way, and it shows the Press and Hold challenge many of you have already met.
We looked at how to recognize a PerimeterX-protected site, using Wappalyzer or a specific cookie. Then we tested free bypass options against Neimanmarcus.com: Undetected Chromedriver and Playwright, running Firefox, Chrome and Brave. We tried them locally and from a datacenter with proxies. We also looked at the behavioral side, with random pauses and scrolling between actions. These setups are a starting point, since every site configures the protection differently.
What’s inside
- How to spot PerimeterX from the Wappalyzer security section or from a cookie in the browser storage.
- Why I used Brave with Undetected Chromedriver instead of Chrome, and what it fixed.
- The Playwright option that slows execution down, and why I think it matters on a low-latency server connection.
- Differences between Playwright with Firefox and with Chromium, and which Chrome settings and persistent context the second one needs.
In the full article: The code of each scraper from the GitHub repository, the Playwright launch options for Firefox, Chrome and Brave, and the per-browser results on the Neimanmarcus category crawl.