Summary of an article published on The Web Scraping Club on 2024-05-17 (for paid subscribers). Read the full article

When I start a scraping project, I check whether the site has an internal API. If it does, that’s the preferred route: less data on both sides, lower cost if proxies are billed per GB, and fewer breakages than with HTML. The trouble starts when the endpoint answers Unauthorized because it wants a Bearer token in the Authorization header. Loading the URL in the browser is not enough.

We first separate Bearer tokens from Castle anti-bot tokens, which are generated from browser data and can’t be replicated by watching the site. Then we work on a real target, the Loewe e-commerce site. In the network tab we trace how the site gets its token: which call issues it, what parameters it needs and where the value feeding that call comes from. From there we build a Scrapy spider that goes from the home page to the product list API.

What’s inside

  • Why Castle’s x-castle-request-token can’t be handled like a Bearer token, and when reverse-engineering it is overkill.
  • The three steps to look for in the network tab: token request, token parsing, final API call.
  • How to find the call that issues the token by searching for the token string in the network tab.
  • How the token refresh and the first token request relate, and why that helps when the scraper enters the site.

In the full article: The Scrapy spider code, the token request parameters, the cookie handling and the sequence of calls from home page to product list API.

Read the full article on The Web Scraping Club