Summary of an article published on The Web Scraping Club on 2025-03-23 (free to read). Read the full article

Akamai Bot Manager sits in front of some of the biggest e-commerce and luxury sites, and a plain Scrapy spider usually gets nowhere with it. Requests hang and the connection is dropped. Before you pick a countermeasure, you need to know Akamai is really there, and that is easy to check.

We cover two ways to detect it: a technology detector like Wappalyzer, and the cookies and network calls you can see in the browser’s dev tools. Then we take Gucci.com as the target. Before reaching for browser automation, I try scrapy-impersonate, the Scrapy plugin built on curl_cffi, and in my experience it works in most Akamai cases. We explain what a TLS fingerprint is, why Akamai looks at it and why Scrapy’s default one gives it away. We also cover how to set up the plugin in a spider and a practical limit when you use it with proxies.

What’s inside

  • Which two Akamai cookies to look for in dev tools, and the other network traces that suggest the sensor script is running.
  • What a TLS fingerprint contains: cipher suites, TLS version, extensions, ALPN and the JA3 hash.
  • What scrapy-impersonate changes compared with standard Scrapy: browser TLS fingerprints and HTTP/2.
  • A caveat on proxies and network errors, and why on Gucci.com not every request needs impersonation once you hold the clearance cookie.

In the full article: The Scrapy settings and request meta for scrapy-impersonate, the spider logs from the Gucci.com test, and the screenshots of Wappalyzer and the Akamai cookies.

Read the full article on The Web Scraping Club