Summary of an article published on The Web Scraping Club on 2024-05-09 (free to read). Read the full article

I started The Web Scraping Club at the end of August 2022. Since then I have published 180 articles, almost all written by me, and today we reached the 50th episode of The Lab series. Web scraping still works like an artisan boutique, with every dataset crafted for a single customer. That problem led us from Re Analytics, our fashion data company, to build Databoutique.com, a marketplace for web-scraped data.

Here I retell my path for newcomers, from C++ scraping in 2009 to today. I explain why I keep publishing anti-bot bypass solutions even though vendors like Cloudflare and Datadome were updating their products long before I started. I also describe what The Lab is: Thursday articles for paying subscribers where I test tools and techniques against anti-bot solutions, with a private repository. I also rank the five most popular Lab episodes.

What’s inside

  • How the Databoutique.com model works: one seller scrapes a site correctly and sells the same dataset many times, using common columns that cover most cases.
  • Why I treat web scraping as a tool and not a black market of data, and how that shapes what I publish.
  • Which Lab episode spent almost a day on the Hacker News front page and what it did to subscriber growth.
  • Which of the top five Lab episodes are now dated, and which one I plan to update.

In the full article: The ranking of the top five Lab episodes with their links, the subscriber growth charts, and my full account of how Re Analytics led to Databoutique.com.

Read the full article on The Web Scraping Club