Summary of an article published on The Web Scraping Club on 2024-02-25 (free to read). Read the full article

Every time we show Databoutique.com to partners and investors, someone asks whether web scraping is dead because of LLMs and AI. Other old objections come up too: that scraping is illegal, that it is easy, that it is expensive. After more than ten years in the industry and thousands of conversations, I think these claims travel by word of mouth more than by facts. If you work in web data, you will be asked the same questions, so it helps to have an answer ready.

I went through each myth from my own experience. For the AI question I used Farfetch as a worked example, looking at how many different datasets a single website can produce and what a generalist model like GPT does with its pages. Then I covered the long tail of website structures, the gap between parsing HTML and crawling, and the legal question around Terms of Service. On cost, I explain why I think the web data market is inefficient and how that led us to build Databoutique.

What’s inside

  • Why one website supports several distinct use cases, using Farfetch as the example with pricing, inventory, product codes and media URLs.
  • Why inventory levels are hard for a generic model: the field labels in the HTML are rarely self-explanatory.
  • Why solving HTML parsing would still leave the work of understanding site structure, finding the right pages and bypassing anti-bot protection.
  • The tomato sauce analogy for a pre-industrial market, and my estimate of how much extraction could be standardized as a commodity.

In the full article: The Farfetch screenshots of the GPT extraction attempts, the Databoutique dataset links, the 50 companies and 100 websites example with its arithmetic, and my full reasoning on the legal and cost questions.

Read the full article on The Web Scraping Club