Summary of an article published on The Web Scraping Club on 2024-11-17 (free to read). Read the full article
Scrapy and Playwright are the two libraries I would pick when starting a Python scraping project from scratch in 2024. Scrapy is the framework, Playwright drives real browsers. Selenium is still popular, but I find Playwright easier and faster for writing a scraper.
This first part covers the Scrapy side. A plain spider is usually enough when the target has no advanced anti-bot and only checks request headers, TLS fingerprinting or the number of requests from one IP. We went through the settings that matter in those cases: default headers and User Agent, cookie handling, request throttling and proxy usage. I also tell a story about old scrapers that stopped working for a reason I did not expect, and how we fixed them. The Playwright configuration for being less detectable comes in part two.
What’s inside
- Why the default Scrapy headers and User Agent flag your requests as a scraper right away, and where to copy realistic ones from.
- How Scrapy passes cookies between requests, and why that matters when a site issues a clearance cookie.
- Two ways to throttle: the Auto Throttle options and a fixed delay, with concurrency limits per domain and per IP.
- Setting a proxy on single requests versus the middleware approach, and what the difference means for proxy costs.
In the full article: The settings.py snippets for headers, cookies, throttling and proxies, the browser Fetch export used to copy headers, and the proxy middleware packages we compare.