Summary of an article published on The Web Scraping Club on 2024-01-25 (for paid subscribers). Read the full article
Kasada is a niche anti-bot with a different approach from the others we have covered. It works like a firewall: it collects hundreds of data points from the browser, applies some AI, and decides whether the request goes through. The first request gets a 429 challenge. If the fingerprint looks like a human browsing, we are redirected to the URL, otherwise we see a white page. Scrapers without a JavaScript rendering engine, like Scrapy, are cut out by default.
Our target is Canada Goose, where we want public prices. We approached it with three tools, starting with undetected-chromedriver. The scraper loads the home page, collects the product categories, scrolls each one to the bottom because of the infinite scroll, then extracts the product data. Along the way I recap what Selenium and webdrivers do and why options like headless mode matter against Kasada.
What’s inside
- Kasada’s core business is fraud prevention, so login and purchase checks are likely stricter. We stay on what is visible when browsing public data.
- Selenium, ChromeDriver, GeckoDriver and EdgeDriver explained, and what undetected-chromedriver changes compared to the standard chromedriver.
- Why the scraper has to handle infinite scroll before extracting anything from a category page.
- The cookie popup on first load and how we collect the category URLs as input for the second part of the scraper.
In the full article: The full scraper code for all three solutions, the browser options as we set them, the category and product extraction logic, and the results of each tool against Kasada.