Summary of an article published on The Web Scraping Club on 2023-03-02 (free to read). Read the full article

The Lab is the series where I cover real-world web scraping use cases, with code and an explanation of the methodology behind each one. It is practical on purpose. Each post starts from a target or a blocker you are likely to meet in production: a Cloudflare, PerimeterX or Datadome protected site, a TLS fingerprint that gets you banned, or data that only lives inside a mobile app.

This page is the archive of the first fourteen posts. Across them we scraped OpenSea NFT data, Airbnb through GraphQL and a Datadome protected API with xsrf tokens. We reverse-engineered a mobile API and changed ciphers in Scrapy. We tried Bezier curves for human-like mouse movements and GoLogin with Playwright against Cloudflare. We also built an anti-detect and anti-bot matrix. On the operations side, we managed fleets of scrapers with Scrapyd and ScrapeOps, and we revisited Cloudflare in an early 2023 update.

What’s inside

  • The series covers three anti-bot vendors on their own: Cloudflare in three separate posts, PerimeterX and Datadome.
  • Some posts focus on technique rather than a target, such as TLS fingerprinting with Scrapy ciphers and Bezier curves for mouse movement.
  • Two posts deal with running scrapers at scale, using Scrapyd and ScrapeOps to schedule and monitor a fleet.
  • Targets include an app, a mobile API, Airbnb’s GraphQL endpoint and OpenSea, so the approach changes from case to case.

In the full article: The code, the methodology and the test setup behind each Lab post are only in the individual articles, which you reach from this archive.

Read the full article on The Web Scraping Club