Summary of an article published on The Web Scraping Club on 2025-07-20 (free to read). Read the full article
Infinite scrolling breaks scraping scripts for two reasons. The content loads dynamically through JavaScript, so you need a browser automation tool. And Playwright, Puppeteer and Selenium don’t give you a simple scroll() method, so you have to simulate scrolling yourself. Even after the scroll works, you still have to wait for the new items to render. Get that wrong and you scrape only part of the data.
I walk through a complete scraping scenario on the ScrapingCourse infinite scrolling e-commerce page, using Playwright with Python. The scroll is simulated with a custom JavaScript script injected into the page. The harder part is the waiting logic, and I show the approach I personally use for it instead of hard waits. I also cover the alternative of intercepting the page’s AJAX requests and why it is more complicated. Then I show how the same logic carries over to Puppeteer in JavaScript and Selenium in Python.
What’s inside
- Why a hard wait is not a real solution and tends to make automation scripts flaky in production.
- Why waiting for the AJAX requests to finish does not tell you the page has finished rendering the content.
- How a predictable number of items per scroll lets you confirm that loading is complete.
- How the scrolling and waiting steps map onto Puppeteer and Selenium, with a Playwright timeout caveat for slower pages.
In the full article: The injected scroll script, the full Playwright code with the selectors and extraction logic, the Puppeteer and Selenium versions, and the sample output from the ScrapingCourse run.