Summary of an article published on The Web Scraping Club on 2022-09-15 (for paid subscribers). Read the full article
Datadome is one of the key players in anti-bot software and runs on sites like FootLocker, Rakuten and Reddit. Sooner or later you will meet it, and a scraper that worked yesterday can stop working today. I had to update one that no longer got past it, so I wrote down the process.
The target is Hermes.com. The product details we need come from an internal API that the site calls while you browse each category. We started with a standard Scrapy spider and adjusted DEFAULT_REQUEST_HEADERS to see if it could get in. Datadome reacted quickly: the spider was redirected and locked out, while the browser, with the same redirect, landed on the home page. We then changed path. Hermes has no official app, so I reused the procedure from the first post of THE LAB to see how the website behaves when accessed from mobile.
What’s inside
- Why the internal API called from the category pages is the data source, and how we spotted it in the browser’s network tab.
- How the plain Scrapy spider is treated by Datadome compared with a real browser hitting the same redirect.
- How the mobile-access method from the first post of THE LAB is applied to a site that has no official app.
- The role of the XSRF tokens in the requests to the Hermes API.
In the full article: The Scrapy spider code, the request headers, the mobile traffic analysis, the handling of the XSRF token and the working scraper for Hermes.com.