Summary of an article published on The Web Scraping Club on 2023-03-02 (free to read). Read the full article
The Lab is the series where I cover real-world web scraping use cases, with code and an explanation of the methodology behind each one. It is practical on purpose. Each post starts from a target or a blocker you are likely to meet in production: a Cloudflare, PerimeterX or Datadome protected site, a TLS fingerprint that gets you banned, or data that only lives inside a mobile app.
This page is the archive of the first fourteen posts. Across them we scraped OpenSea NFT data, Airbnb through GraphQL and a Datadome protected API with xsrf tokens. We reverse-engineered a mobile API and changed ciphers in Scrapy. We tried Bezier curves for human-like mouse movements and GoLogin with Playwright against Cloudflare. We also built an anti-detect and anti-bot matrix. On the operations side, we managed fleets of scrapers with Scrapyd and ScrapeOps, and we revisited Cloudflare in an early 2023 update.
What’s inside
- The series covers three anti-bot vendors on their own: Cloudflare in three separate posts, PerimeterX and Datadome.
- Some posts focus on technique rather than a target, such as TLS fingerprinting with Scrapy ciphers and Bezier curves for mouse movement.
- Two posts deal with running scrapers at scale, using Scrapyd and ScrapeOps to schedule and monitor a fleet.
- Targets include an app, a mobile API, Airbnb’s GraphQL endpoint and OpenSea, so the approach changes from case to case.
In the full article: The code, the methodology and the test setup behind each Lab post are only in the individual articles, which you reach from this archive.