Summary of an article published on The Web Scraping Club on 2024-03-24 (free to read). Read the full article
Ten years ago Andrea Squatrito and I started scraping real estate websites. Our setup was one Hetzner server, C++ with cURL, no proxies and no rotating IPs. It was enough to run 50+ websites at once and build a large database of listings from around the world. Number of sales: zero. The tech stack and the targets changed over the years, but selling web data stayed hard.
At Re Analytics, our data factory company, we tried two markets: alternative data for finance and web data for the retail industry, mostly fashion. For each one I write a simple equation of what it takes to sell. I also give my view on whether web scraping is still worth doing, why we turned Re Analytics into Databoutique.com, a marketplace for web scraped data, and how that model changes the equations for sellers and buyers.
What’s inside
- What a data vendor needs to sell alternative data to funds: accuracy, data history and exclusivity, and why exclusivity helps your customers but not your business.
- The risk of collecting data for a year or two, then finding no correlation with stock prices once analysts backtest it.
- Why retail customers each want different websites and focus areas, and the Zara versus tailor comparison.
- Why almost every internal scraping project fails while market solutions stay too expensive, which leaves many potential customers unserved.
In the full article: Equations for each market, the reasoning behind the Databoutique pivot, and the seller and buyer sides of the marketplace model, written out in full.