Summary of an article published on The Web Scraping Club on 2023-03-07 (free to read). Read the full article
Most tools we use in web scraping were built for browser automation and testing, not for scraping. Selenium is the typical case. Anti-bot solutions can tell when they are facing a scraper driven by a Selenium webdriver, and that is a problem when you need to get past them.
For this reason the scraping community built a webdriver designed for the job, Undetected Chromedriver. Here I explain how it differs from the Selenium Chromedriver. In short, it is a webdriver whose fingerprint and settings are comparable to a browser controlled by a human. I then show how simple it is to use. You install the package with pip and import it into a Python program. A short video tutorial goes through the usage in more detail.
What’s inside
- Why anti-bot solutions can recognize a scraper running on a standard Selenium webdriver
- What it means for a webdriver to have a fingerprint and settings comparable to a human-controlled browser
- Installation is a single pip package, used from Python
- A video tutorial walks through the usage in more detail
In the full article: Code of the usage example, the GitHub project page screenshot and the embedded video tutorial on using Undetected Chromedriver.