Summary of an article published on The Web Scraping Club on 2022-11-13 (free to read). Read the full article
Selenium and Playwright come up every time a Python scraper needs a fully headed browser. It is almost ironic that both were built for something else: helping front-end developers test their sites across browsers. A scraper is an automated browser going around the web, so the fit is natural, but the two tools get there with very different architectures, and that matters once a project grows.
I go through what each tool is and how it talks to the browser. For Selenium, that means webdrivers installed separately and the path a command takes from the scraper to the browser. For Playwright, it means one API over Chromium, Firefox and WebKit, with bundled browsers and no webdriver. I leave Puppeteer out, since I prefer Python and Pyppeteer is an unofficial port I haven’t tried. Then I say which of the two I choose for large scraping projects and why, based on setup, maintenance and anti-bot tooling.
What’s inside
- Why Selenium needs a separate webdriver for each browser, and how a command travels from the scraper to the browser.
- How Playwright differs: bundled, slightly modified browsers driven directly through an API, while still allowing a standard Chrome.
- Why Puppeteer is out of scope: JavaScript only, and Pyppeteer is unofficial and untested by me.
- How Playwright handles stealth plugins and persistent contexts with a real Chrome user profile.
In the full article: The links to the Checkly benchmark, the Scrapfly guide with Playwright code, and the full reasoning behind my choice, including the persistent context setup.