Summary of an article published on The Web Scraping Club on 2022-08-21 (free to read). Read the full article
Scrapy covers most of a Python scraping job, but not all of it. It reads only static HTML, proxies need managing, some targets need a real browser, and you want to know what a site runs before you write a spider. A starter toolkit should fill these gaps without piling up tools.
I go through the tools we use. First Scrapy and its modular architecture. Then Advanced Scrapy Proxies, a package I wrote and we run daily in production. Then scrapy-splash, which routes requests to a Splash server so JavaScript gets executed. Then Microsoft Playwright with playwright-stealth, for sites protected by anti-bot software. Last is the Python wrapper for Wappalyzer, which identifies the technology stack behind a target. I also ask for suggestions on what is missing from the list.
What’s inside
- What Downloader Middlewares can do to requests and responses, for example filtering duplicated requests before they reach the website.
- How Item Pipelines handle the output of a spider, and why that suits structured data with several columns per row.
- Where the proxy package can read its list from: a public URL, a local file, or a proxy set directly in the options.
- How Playwright compares with Selenium, and what Splash is built on.
In the full article: The Scrapy architecture diagram, the request and response workflow between its components, and the links to the documentation and repository of each tool.