Summary of an article published on The Web Scraping Club on 2024-03-24 (free to read). Read the full article

Ten years ago Andrea Squatrito and I started scraping real estate websites. Our setup was one Hetzner server, C++ with cURL, no proxies and no rotating IPs. It was enough to run 50+ websites at once and build a large database of listings from around the world. Number of sales: zero. The tech stack and the targets changed over the years, but selling web data stayed hard.

At Re Analytics, our data factory company, we tried two markets: alternative data for finance and web data for the retail industry, mostly fashion. For each one I write a simple equation of what it takes to sell. I also give my view on whether web scraping is still worth doing, why we turned Re Analytics into Databoutique.com, a marketplace for web scraped data, and how that model changes the equations for sellers and buyers.

What’s inside

  • What a data vendor needs to sell alternative data to funds: accuracy, data history and exclusivity, and why exclusivity helps your customers but not your business.
  • The risk of collecting data for a year or two, then finding no correlation with stock prices once analysts backtest it.
  • Why retail customers each want different websites and focus areas, and the Zara versus tailor comparison.
  • Why almost every internal scraping project fails while market solutions stay too expensive, which leaves many potential customers unserved.

In the full article: Equations for each market, the reasoning behind the Databoutique pivot, and the seller and buyer sides of the marketplace model, written out in full.

Read the full article on The Web Scraping Club