Summary of an article published on The Web Scraping Club on 2023-05-28 (free to read). Read the full article

Scale kills data quality, in data teams and in web scraping alike. You start with a few scrapers, no stored logs, no standards and no quality checks. A dozen websites later, scraping continuously and accurately needs a real ingestion and quality control process, and you have no source of truth to check against. That is the thread running through this May recap.

I went through the Substack posts I read this month and picked three. Chad Sanderson on how scale kills data teams, which I read as a former DWH analyst. Andrea Squatrito on QA for in-house scraping, which leads me to ROI. Jason DeRise on how finance buyers evaluate data partners. I also name the most-read post of the month, which is about AI for the third month in a row.

What’s inside

  • Why ELT without a single source of truth leaves departments with duplicated rules and the same KPI calculated with different formulas.
  • Three techniques to limit data quality issues in in-house scraping: time series, manual counts on the target site and redundancy with other providers, each with its limits.
  • Why a near real-time web data feed with 100% uptime is a utopia, even for a single website.
  • What a finance buyer asks a data provider: ethical collection, backtesting and how fast it adapts to feedback.

In the full article: Links to the three Substack posts and to the extra Data Boutique piece on the ROI of scraping projects, plus the most-read post of the month.

Read the full article on The Web Scraping Club