Summary of an article published on The Web Scraping Club on 2023-03-03 (free to read). Read the full article
If you are looking for a starting point on web scraping tools, this page is not a single tutorial. It is a tutorials index for The Web Scraping Club, and it collects the posts I wrote, plus one by Fabien Vauchelles, over the first years of the newsletter. The topics are the ones we deal with daily: picking a browser automation tool, getting past anti-bot systems, handling proxies, and keeping many scrapers running without a huge team.
The list runs from basics to field work. On the basics side we have a first Scrapy spider, HTTP requests in Python, what a proxy server is, and Selenium against Playwright. On the practical side we have bypassing Cloudflare, the 429 error served by Kasada, antidetect browsers, reverse-engineering a mobile API, a home made mobile proxy and scraping Ikea sites. We also cover scheduling fleets with Scrapyd and ScrapeOps, the costs of web scraping, and how we scaled Databoutique.com to 2 billion prices a month.
What’s inside
- A pre-coding checklist for target analysis, starting with identifying the technology stack and known anti-bot solutions on the website.
- Two ways to run a fleet of scrapers: Scrapyd for scheduling and ScrapeOps for managing executions.
- A look at costs and difficulty: cheap cloud computing opened up scraping at scale, yet many of us feel it is getting harder and pricier.
- Opinion pieces sit among the technical posts, including a roundup of GitHub repositories from 2023 and a discussion on whether AI is stealing our jobs.
In the full article: Each tutorial itself, with its code, test results, configurations and the reasoning behind the tool choices, is only available by opening the individual posts.