Summary of an article published on The Web Scraping Club on 2024-10-13 (free to read). Read the full article
I have been in web scraping since 2014, and the question I keep coming back to is whether it is a profitable business. Selling web data or services built on it does not scale: more customers means more people to hire, and margins stay low. Everyone in the same industry also scrapes the same websites, fights the same anti-bots, and ends up with the same prices, reviews or locations.
I go through three models we know firsthand or from peers: alternative data sold to hedge funds, SaaS APIs that monitor popular websites, and market intelligence built on full-site scraping, which is what we do at Re Analytics. For each I look at where growth stalls. Then I argue that web data is a commodity and can be bought rather than extracted by every company. That is the idea behind Data Boutique, the marketplace we started and now use ourselves as both sellers and buyers. I close with who actually makes money in this industry.
What’s inside
- Why hedge funds ask for a multi-year backtest and a due diligence questionnaire before buying a dataset, and how exclusivity affects price.
- What happened to our scraping scope at Re Analytics each time we signed a new customer in luxury e-commerce.
- Why per-request SaaS monitoring is hard to reuse across customers, even though its scraping costs track revenue.
- The comparison with cloud computing and in-house servers, and what I would do first if I started a web data company today.
In the full article: The full numbers on how our website coverage grew per customer, the three-step diagram of how data companies build on the same sources, and my closing answer on who profits in web scraping.