Summary of an article published on The Web Scraping Club on 2025-11-16 (free to read). Read the full article
Browser automation for scraping usually means a WebDriver to keep in sync with the browser, and clicks and scrolls that anti-bot systems spot quickly. Pydoll is an open source Python library that skips the driver and talks to Chrome directly over the Chrome DevTools Protocol. It is async-first and built for JavaScript-heavy sites, and it also works for end-to-end testing.
I introduce what the library is and what it offers, then build a scraper with it. The target is the infinite-scrolling e-commerce demo page from Scraping Course, which needs JavaScript execution and scrolling to load its products. We cover installation, navigation, scrolling, waiting for elements and parsing the product data. Then I go through the API: element finding with find() and query(), and click interactions meant to look less mechanical. I also point out what the project does not do yet, notably its human-like interaction engine, which is still announced for future releases.
What’s inside
- The difference between find() for attribute searches and query() for CSS and XPath selectors, and how to handle optional elements without exceptions.
- Why Pydoll’s Scroll API waits for the browser’s scroll to finish, while a plain JavaScript scrollBy returns immediately.
- Where human-like scrolling stands today: the built-in API is not available yet, and a helper function from the maintainer partly fills the gap.
- How clicks can be varied with position offsets and hold time, and how they differ from JavaScript-based clicks.
In the full article: The complete Pydoll scraper script with its output, the exact scrolling and waiting logic, the click and element-finding code examples, and the remaining sections on the API and the project roadmap.