Summary of an article published on The Web Scraping Club on 2022-11-13 (free to read). Read the full article

Selenium and Playwright come up every time a Python scraper needs a fully headed browser. It is almost ironic that both were built for something else: helping front-end developers test their sites across browsers. A scraper is an automated browser going around the web, so the fit is natural, but the two tools get there with very different architectures, and that matters once a project grows.

I go through what each tool is and how it talks to the browser. For Selenium, that means webdrivers installed separately and the path a command takes from the scraper to the browser. For Playwright, it means one API over Chromium, Firefox and WebKit, with bundled browsers and no webdriver. I leave Puppeteer out, since I prefer Python and Pyppeteer is an unofficial port I haven’t tried. Then I say which of the two I choose for large scraping projects and why, based on setup, maintenance and anti-bot tooling.

What’s inside

  • Why Selenium needs a separate webdriver for each browser, and how a command travels from the scraper to the browser.
  • How Playwright differs: bundled, slightly modified browsers driven directly through an API, while still allowing a standard Chrome.
  • Why Puppeteer is out of scope: JavaScript only, and Pyppeteer is unofficial and untested by me.
  • How Playwright handles stealth plugins and persistent contexts with a real Chrome user profile.

In the full article: The links to the Checkly benchmark, the Scrapfly guide with Playwright code, and the full reasoning behind my choice, including the persistent context setup.

Read the full article on The Web Scraping Club