Summary of an article published on The Web Scraping Club on 2023-02-05 (free to read). Read the full article

Python scraping advice is scattered, and most of it stops at the first block. I put the posts I wrote on the topic in one place, ordered around the problems you hit in practice: getting a first spider running, choosing a browser automation tool, and getting past anti-bot systems on real targets.

The list starts with a first Scrapy spider and a Selenium vs Playwright comparison. Then comes THE LAB, the series with code and methodology on real cases: scraping data from an app, Datadome with xsrf tokens, Cloudflare, PerimeterX, and Airbnb through GraphQL. Other posts cover changing ciphers in Scrapy against TLS fingerprinting, Bezier curves for human-like mouse movements, and Cloudflare with GoLogin. There is also an anti-detect and anti-bot matrix, a selection of GitHub repositories from 2023, and two data projects: the Kallax Index on Ikea and OpenSea NFT data.

What’s inside

  • Two entry points for beginners: a first Scrapy spider and a Selenium vs Playwright comparison.
  • Cloudflare appears twice in THE LAB, once with a standard approach and once with GoLogin.
  • Besides anti-bot posts, there are two data projects: the Kallax Index on Ikea and OpenSea NFT data.
  • Anti-detect tools are matched against anti-bot systems in a matrix, and a separate post lists the GitHub repositories worth following in 2023.

In the full article: Links to each tutorial with its opening paragraph, which shows how each post frames its target and the tools it uses.

Read the full article on The Web Scraping Club