Summary of an article published on The Web Scraping Club on 2024-02-29 (for paid subscribers). Read the full article
Inventory data is like looking through the keyhole of a competitor’s warehouse. Take a multi-brand clothing store that already has price monitoring. It still cannot tell whether a competitor holding the same sweater, in another color, struggles to sell it too. A fast fashion brand has the mirror problem: it wants the top sellers in a given geography while a trend is just starting. Price data answers neither question.
We go through what a stock tracking project needs: the data must be used on the site, and collecting it must not clash with the terms of service. Then we cover how to choose the scraping frequency, from car dealers to groceries, and how to interpret the numbers, with a fashion site and the Lowe’s case from The Lab #28. We also list the main challenges, which are heavier than in price monitoring.
What’s inside
- Why the scraping frequency depends on how often people buy a product, and how it trades estimate accuracy against cost.
- How to work out what a stock number refers to, such as a single store or a wider warehouse, by comparing the same product across locations.
- Why a useful history takes months or years, with two years as a minimum for a dataset aimed at financial investors.
- The terms-of-service check when stock only appears after a login or after adding an item to the cart.
In the full article: Common places where inventory data can be found on a website, which is the part this page leaves out.