Summary of an article published on The Web Scraping Club on 2023-12-03 (free to read). Read the full article

Once the technical audit and the legal check are done, you have to pick the language and the tools. Python and Node.js are the two that matter in scraping. Choosing badly costs time. A scraper can run fine on your laptop and then fail once deployed on the target infrastructure, or die after the first few requests.

This is a lesson of the free Web Scraping from 0 to Hero course. I explain why I stay on Python after 10+ years in the field, and when Node.js might have an edge. Then I go through the tools I suggest to start with. On the Python side that means Scrapy and Playwright, split by whether the target has anti-bot protection. On the Node.js side I cover Axios, Cheerio and Puppeteer. That section comes from what I’ve read, not from my own projects. I also say why choosing the tool is only the first part of a project that is more complex than it looks.

What’s inside

  • Why I skip the legal side in the course and what to ask a professional before you collect copyrighted or personal data
  • The one case where I’d lean toward Node.js over Python, and why I can’t call either language the best
  • Scrapy’s origins, its current maintainer, and the plugins that extend it for JavaScript rendering, distributed runs and scheduling
  • Puppeteer’s relation to Playwright and the language limit that separates them

In the full article: The plugin list for Scrapy, the feature breakdowns of Axios and Cheerio, the course format and tag, and the link to the next episode on coding a first Scrapy scraper.

Read the full article on The Web Scraping Club