Summary of an article published on The Web Scraping Club on 2024-01-21 (free to read). Read the full article

Scrapy is our default starting point, but it only reads the raw HTML. When the data shows up only after JavaScript runs, or after a click, a form or a menu, it stops being enough. Anti-bot challenges that need a JS engine are another case where it falls short. That is where a browser automation tool enters the toolbelt, and Microsoft Playwright is the one we’re adding to the Web Scraping from 0 to Hero course.

I cover where Playwright comes from, its link to Puppeteer, and why Python users like me can use it where Puppeteer is JavaScript only. Then I go through its core features, from the patched browsers it ships with to headless mode, auto-wait, network interception and mobile emulation. I also say plainly what it costs: more resources, slower runs, and scrapers that in my experience can degrade over time. The last part compares it with Scrapy and gives a simple rule for choosing between them.

What’s inside

  • How Playwright relates to Puppeteer, and why only one of them fits a Python-based course.
  • Playwright ships with patched browsers, but you can also launch your own installed Chromium-based browser such as Chrome or Brave.
  • Network interception has a practical use with proxies: blocking images so you don’t waste bandwidth and money.
  • A quick check using “view source” versus “Inspect” to decide whether Scrapy is enough or you need a browser.

In the full article: The full comparison of Playwright and Scrapy across dynamic pages, browser-specific rendering, anti-bot evasion and speed, plus the diagram summarizing it and the link to the course archive.

Read the full article on The Web Scraping Club