Summary of an article published on The Web Scraping Club on 2022-08-05 (free to read). Read the full article
Looking for a parka on a brand’s e-commerce site, I saw a 429 “Too many requests” on the home page and an unusual response header. It was Kasada, one of the newest anti-bot vendors. Most solutions look for fingerprint red flags or monitor user behavior. Kasada treats every first request as a bot until proven otherwise. That is a strong choice, and it shows in the site’s visibility on Google, DuckDuckGo, Bing and Baidu.
I took it as a challenge and started with Playwright plus the Stealth plugin on Chrome, not Chromium. Then I went back to the SEO angle and read the site’s robots.txt, which names one specific crawler. I looked at what that tool is and what its presence says about how the protection is set up. I also ask whether fighting bots this aggressively is worth the risk, and what public data access could change for market intelligence scrapers.
What’s inside
- How Kasada’s “guilty until proven innocent” model differs from the fingerprint-check and behavior-monitoring approaches of other anti-bots.
- An SEO check on four search engines, including a Bing ranking behind a similarly named brand and the Chinese version missing from Baidu’s first page.
- What we do not know about the trade-off: how many bots hit the site before Kasada, what the solution costs, and how many sales the SEO gap loses.
- The case for exposing public catalog data through an API, so anti-bot tools can focus on fraud and DDoS.
In the full article: Screenshots of the 429 response, the header, the search results per engine, the failed Playwright run and the robots.txt entry, plus the change that got past the block.