Summary of an article published on The Web Scraping Club on 2022-08-05 (free to read). Read the full article

Looking for a parka on a brand’s e-commerce site, I saw a 429 “Too many requests” on the home page and an unusual response header. It was Kasada, one of the newest anti-bot vendors. Most solutions look for fingerprint red flags or monitor user behavior. Kasada treats every first request as a bot until proven otherwise. That is a strong choice, and it shows in the site’s visibility on Google, DuckDuckGo, Bing and Baidu.

I took it as a challenge and started with Playwright plus the Stealth plugin on Chrome, not Chromium. Then I went back to the SEO angle and read the site’s robots.txt, which names one specific crawler. I looked at what that tool is and what its presence says about how the protection is set up. I also ask whether fighting bots this aggressively is worth the risk, and what public data access could change for market intelligence scrapers.

What’s inside

  • How Kasada’s “guilty until proven innocent” model differs from the fingerprint-check and behavior-monitoring approaches of other anti-bots.
  • An SEO check on four search engines, including a Bing ranking behind a similarly named brand and the Chinese version missing from Baidu’s first page.
  • What we do not know about the trade-off: how many bots hit the site before Kasada, what the solution costs, and how many sales the SEO gap loses.
  • The case for exposing public catalog data through an API, so anti-bot tools can focus on fraud and DDoS.

In the full article: Screenshots of the 429 response, the header, the search results per engine, the failed Playwright run and the robots.txt entry, plus the change that got past the block.

Read the full article on The Web Scraping Club