Summary of an article published on The Web Scraping Club on 2023-03-03 (free to read). Read the full article

If you are looking for a starting point on web scraping tools, this page is not a single tutorial. It is a tutorials index for The Web Scraping Club, and it collects the posts I wrote, plus one by Fabien Vauchelles, over the first years of the newsletter. The topics are the ones we deal with daily: picking a browser automation tool, getting past anti-bot systems, handling proxies, and keeping many scrapers running without a huge team.

The list runs from basics to field work. On the basics side we have a first Scrapy spider, HTTP requests in Python, what a proxy server is, and Selenium against Playwright. On the practical side we have bypassing Cloudflare, the 429 error served by Kasada, antidetect browsers, reverse-engineering a mobile API, a home made mobile proxy and scraping Ikea sites. We also cover scheduling fleets with Scrapyd and ScrapeOps, the costs of web scraping, and how we scaled Databoutique.com to 2 billion prices a month.

What’s inside

  • A pre-coding checklist for target analysis, starting with identifying the technology stack and known anti-bot solutions on the website.
  • Two ways to run a fleet of scrapers: Scrapyd for scheduling and ScrapeOps for managing executions.
  • A look at costs and difficulty: cheap cloud computing opened up scraping at scale, yet many of us feel it is getting harder and pricier.
  • Opinion pieces sit among the technical posts, including a roundup of GitHub repositories from 2023 and a discussion on whether AI is stealing our jobs.

In the full article: Each tutorial itself, with its code, test results, configurations and the reasoning behind the tool choices, is only available by opening the individual posts.

Read the full article on The Web Scraping Club