What it is
This project presents four efficient techniques designed to detect bots that utilize Selenium running in both headless and non-headless Chrome environments. These methods are effective for identifying automated browser instances.
How it works
The detection techniques rely on analyzing various browser and JavaScript artifacts. One method involves checking the user agent HTTP headers or the navigator.userAgent in JavaScript to identify user agents linked to Headless Chrome, such as those containing HeadlessChrome/126.0.6478.114.
Other techniques include detecting the presence of the HeadlessChrome substring within the sec-ch-ua header, checking if navigator.webdriver is set to true in JavaScript, and detecting side effects resulting from the Chrome DevTools Protocol (CDP).
TWSC experience
Not yet tested by TWSC.
Related
Sources
The Web Scraping Wiki is compiled from the tests and research published on The Web Scraping Club, the newsletter for web scraping professionals. Subscribe to get the next article in your inbox.