Summary of an article published on The Web Scraping Club on 2023-02-06 (free to read). Read the full article
Selenium is usually introduced as a testing tool, but for those of us in web scraping it is mostly a way to drive a real browser. It automates the repetitive parts of a session: filling out forms, clicking buttons, navigating pages and pulling data out of them. It works with several languages, including Java, Python and Ruby, so it fits most teams’ existing stack.
Here I give a basic introduction to what Selenium is and how it is built. I go through its three components: WebDriver, which talks to the browser, Grid, which runs tests in parallel across browsers and machines, and IDE, a browser extension that records and replays actions. Then I look at how they apply to scraping, with the general flow from installing a WebDriver for your language to using the extracted data. I also list the advantages over traditional scraping methods, such as handling dynamic pages.
What’s inside
- What each of the three Selenium components does and which one matters most for scraping
- How Selenium Grid runs tests in parallel on different browsers and machines
- The IDE extension as a record-and-replay option for building automated tests
- The benefits claimed for Selenium over traditional scraping methods, including dynamic pages and repetitive tasks
In the full article: The full list of scraping steps with WebDriver and the complete comparison of Selenium against traditional scraping methods.