Summary of an article published on The Web Scraping Club on 2024-05-12 (free to read). Read the full article

Getting the text out of the HTML is not the end of the job. A price can come with or without a thousands separator, with a comma or a dot, with or without a currency symbol. Descriptions carry new line characters, and country and currency codes change from site to site. If you want to compare extractions, you need a data cleaning and standardization process.

There are two places to put it: in the scraper, or in a database after loading. We walk through a data quality process for structured e-commerce data, from numeric and string cleaning to standardization and validation, up to publishing to the final user. We show it on a SQL database, using regular expressions and lookup tables for country codes. Then we look at the same steps embedded in a Python scraper, with a library like Price-Parser. We close by weighing the two approaches, especially when the number of scrapers grows.

What’s inside

  • How a lookup table with two columns keeps country standardization maintainable, since the values you meet are many but finite.
  • Which validation checks to run before publishing: e-mail format, valid product URLs, empty fields, unmapped values, and how to link them to alerts.
  • Why standardization is the harder step inside the scraper, and the options for bringing the target values in.
  • The cost of adding a database to your stack when you only scrape a few websites.

In the full article: The full step-by-step pipeline diagram, the lookup table example, the data quality checklist and the complete pros and cons comparison of the two approaches are only in the full article.

Read the full article on The Web Scraping Club