Summary of an article published on The Web Scraping Club on 2022-11-27 (free to read). Read the full article
Public data is only useful if someone can actually get it. Government datasets often sit behind FOIA requests that take so long the journalist no longer needs them, and they arrive with quality and cleaning problems. Jeremy Singer-Vine, a data journalist in New York, started the Data Liberation Project to fix that. Web scraping is also becoming a way to check what governments say, and a similar case came from Turkey.
This month’s recap covers two stories. The first is the Data Liberation Project: how Singer-Vine combines FOIA requests with scraping to build clean, reusable datasets for reporters, and why he skipped the midterms for slower, broader projects. I also cover the Turkish researchers who scraped online retailers daily to test official inflation figures. The second story is Oxylabs, a finalist at the Baltic Sustainability Awards for an AI-powered scraping tool built with Lithuania’s communications regulator to find illegal images. I close with a short comment on how technology is neither good nor bad.
What’s inside
- How Singer-Vine splits the work between FOIA requests, at a pace of about five a month, and scraping of public sources that are hard to use.
- Why he chose long-term government tracking over midterm election data, and where the idea came from during his time as data editor at BuzzFeed.
- The Turkish inflation case, where daily scraping of online retailers was set against the figure the government reported.
- How the Oxylabs tool scans the Lithuanian IP address space and passes what it finds to the regulator’s hotline for human review.
In the full article: Links to the original sources (CJR, the Financial Times report, Accesswire), the Oxylabs quote on the project, and the figures from the Turkish inflation study.