Summary of an article published on The Web Scraping Club on 2026-06-21 (free to read). Read the full article
A competitive intelligence scraper can run for weeks with a green dashboard and still return empty strings. Someone asks whether a competitor changed pricing last week, and you have no answer and no record of what the page looked like before. The usual causes are structural drift, soft blocks, schema rot, and timezone or locale traps. The defensive code we add to avoid crashes is often what turns a broken scraper into a quiet one.
We built the pipeline around a monitoring layer and a storage layer. The monitoring layer has four parts: output validation, structural fingerprinting, data freshness checks, and canary fields. We use Python with httpx, BeautifulSoup and Pydantic, and run the validation against Stripe’s pricing page. Storage keeps two layers, raw and processed. S3 holds the raw HTML, and a SQL table holds the structured records that dashboards and alerts read.
What’s inside
- Why a scraper that runs from the EU and one that runs from the US can both pass validation on the same Stripe pricing page while tracking different currencies
- Why the DOM fingerprint hashes tag and class names and ignores text content, so price changes don’t raise false alarms
- The limit of selector-based fingerprinting, and the options to reduce it: several selectors, a whole-body check, or an LLM-based approach
- How each processed record links back to the exact raw HTML it came from, and carries the page fingerprint recorded at scrape time
In the full article: The full Pydantic validator, the fingerprint and canary check code, the S3 and SQL storage code, the Stripe output from the EU and the US, and the rest of the pipeline.