Summary of an article published on The Web Scraping Club on 2023-06-15 (free to read). Read the full article
Brands care about price consistency. The same pair of shoes at different prices in different places undermines the price a brand declares, and the effect is stronger in luxury. Online, customers compare across the brand site, marketplaces and third-party retailers, so monitoring those prices is a real business need and a typical web scraping use case. Nike, which pushed direct-to-consumer and now also works with selected retailers, is our example.
We built a small price monitoring tool around the Nike Air Jordan 1 Mid, using Nimble as the data layer. First we set a baseline by scraping Nike’s own site in several countries and colors with the Nimble Browser inside a Scrapy project, then normalized prices to USD to look at geo pricing. Then we moved to wholesalers, using the Nimble E-commerce API on Walmart search results and its marketplace sellers. I also say what I think of both products and where they still need work.
What’s inside
- Why the Nike site is harder than the scraper: some countries put all colors on one page, Canada has one page per color, and Saudi Arabia uses different product IDs.
- Where the schema.org data on the product page was enough, and why we parsed what the API returned.
- A proxy authentication quirk with the Nimble Browser that needed a custom Scrapy middleware.
- How the Amazon side would work, and what I saw when I tried the E-commerce API outside the US.
In the full article: The Scrapy project and the requests script on GitHub, the geo pricing table by country, the Walmart seller output and my full assessment of the Browser and E-commerce APIs.