Summary of an article published on The Web Scraping Club on 2023-03-18 (free to read). Read the full article
When I think of scraping an e-commerce site, I think of prices, because that is what I scraped for years. It is only a small part of what these sites hold. Which data matters most depends on the industry of the target: promotions, product features, reviews, availability, positioning and brand distribution all have their use.
This is a 101, so we start from the basics. We list the data types you can extract and set the ground rules for ethical scraping that make it legal. Then we fix the glossary: product list page (PLP), product detail page (PDP), and the three distribution models, monobrand websites, multibrand websites and marketplaces. We use the library analogy to explain PLP and PDP, and say whether a scraper needs only the list or also the detail pages. Each website type brings different challenges, from regional versions to anti-bot protection to sheer size and cost, and those challenges shape how you plan the scraper.
What’s inside
- Six types of e-commerce data besides price, from reviews and availability to positioning and brand distribution, and what each is used for
- The ethical scraping rules we follow, including the clickwrap Terms of Service caveat and the preference for APIs
- How the choice between scraping only PLPs or also PDPs depends on the data you need
- Why monobrand, multibrand and marketplace sites each need a different approach, with Chanel and Amazon as examples
In the full article: Screenshots of a real PLP and PDP, the full list of ethical scraping rules and the examples used for each website type are only in the full article.