Summary of an article published on The Web Scraping Club on 2026-10-25 (free to read). Read the full article

An empty extraction does not prove a page is JavaScript-rendered. The selector may be wrong, your client may have followed a redirect without you noticing, or the server may have served a CAPTCHA page in place of the content. Reaching for a browser every time is slower and costlier, and it often fixes the wrong problem.

We start from the difference between a headless browser and a plain HTTP request. Then we go through three symptoms: skeletal HTML, content hidden behind user interaction, and bot defenses. We check them with requests and BeautifulSoup on Quotes to Scrape, a scrapingcourse.com page and an L’Occitane product page. Next come the quick tests to run before committing to a browser: inspecting elements, disabling JavaScript and reloading, and reading the Network tab on a Swiss real estate site. The last step is choosing between Puppeteer, Playwright and Selenium based on the shape of your workload.

What’s inside

  • Missing elements do not always mean missing data: some pages embed their records in script tags or serialized application state, so you may extract without rendering.
  • Multi-step flows such as logins, date pickers and filters depend on cookies and values from earlier actions, so opening the page in a headless browser does not complete them.
  • A headless browser does not guarantee you get past a CAPTCHA wall, but it is useful for debugging, for example with a screenshot of what the browser actually receives.
  • In the Network tab, APIs external to the site and APIs internal to it (or encrypted) lead to different choices, and intercepting the external ones is sometimes the easier path.

In the full article: Python code for each test, the raw HTML and screenshots from every target page, the Network tab walkthrough, and the comparison of Puppeteer, Playwright and Selenium.

Read the full article on The Web Scraping Club