Summary of an article published on The Web Scraping Club on 2024-07-11 (for paid subscribers). Read the full article

PerimeterX, now part of HUMAN Security, sits with Cloudflare, Datadome and Kasada among the main anti-bot solutions. It protects sites like Crunchbase, Zillow and SSense. Before writing a scraper you need to know what you are facing: Sensor, Detector and Enforcer, the Press and Hold challenge or ReCAPTCHA on top, and Hype Sale for sensitive areas like checkout.

We start by recognizing it from its cookies and the domains it calls in the network tab. Then we pick a target protected only by PerimeterX 3, with the _px3 cookie and no other anti-bot, and that target is Crunchbase. We stay on public pages and leave the logged-in content alone. We look at how the site behaves in an incognito browser, then test a Scrapy spider with scrapy-impersonate and Playwright with Chrome on a local Mac, from a clean home IP.

What’s inside

  • How to check a site for PerimeterX with Wappalyzer, and why its result needs a second check.
  • Why Hype Sale uses slower, heavier challenges and can only cover a small part of a website.
  • Why Crunchbase was chosen as the test target, and where we draw the line on paywalled data and Terms of Service.
  • How the order of page visits in a manual incognito session shapes the design of the scraper.

In the full article: The Scrapy spider and Playwright code, the GitHub folder 56.PERIMETERX3, the second Playwright attempt and the working setup for Crunchbase are only in the full article.

Read the full article on The Web Scraping Club