Summary of an article published on The Web Scraping Club on 2026-05-24 (free to read). Read the full article

“Open data” sounds like a green light, but it is not. A dataset can be public and still be restricted by its license, full of personal data, or hosted on a portal that was never built to take a crawler. Most real targets sit in the grey zone: government portals, court records, municipal datasets with no explicit license, where default copyright applies. Add GDPR reach over EU residents and the risk of combining clean datasets into profiles of real people, and the problem is bigger than the scraper.

We built a four-step framework that puts the checks before the code: license first, API or bulk download before scraping, stricter politeness on weak infrastructure, and PII handling at collection time. Then we turned two parts into Python. One queries the CKAN metadata API to read a dataset’s license before downloading anything, with fallbacks for portals that don’t run CKAN. The other is a field-level PII audit for structured records, plus what to do when you have no schema.

What’s inside

  • How the main licenses (CC0, CC-BY, CC-BY-SA, CC-BY-NC, ODbL) change what you can redistribute or sell, and what to do when no license is stated.
  • Why hammering a municipal or university portal is arguably worse than hitting a big e-commerce site, and the extra rules we apply: strict Crawl-delay, local caching, off-peak runs, collecting only what you need.
  • A split of PII into direct identifiers, quasi-identifiers and GDPR sensitive categories, with the choice between pseudonymization and anonymization.
  • Where field-level detection fits, where NER with spaCy falls short on unstructured text, and why LLM-based structuring is an alternative.

In the full article: The Python code for the CKAN license check and its sample output, the full field-level PII audit pipeline, the license summary table and the fallback checks for non-CKAN portals.

Read the full article on The Web Scraping Club