Summary of an article published on The Web Scraping Club on 2023-06-18 (free to read). Read the full article

LLMs are trained on huge amounts of data, and the largest source of that data is the Internet. That makes web scraping the shovel in the AI gold rush. Reddit’s API pricing change, which pushed thousands of subreddits private, shows what happens when a data owner realizes it has been gifting shovels to the miners. Scraping is still seen as easy by people who built one project, but at scale it is human-intensive and anti-bot costs keep rising.

I start the month of AI on The Web Scraping Club, in collaboration with Nimble, with a look at where the industry stands. We cover the gap that remains even with good anti-bot bypass tools: you still write hundreds of scrapers for hundreds of websites. We look at AI services trained on website HTML that try to unblock and parse at the same time, and what we have seen from them in the field. Then I describe the market inefficiency I have seen in a decade in the industry, and why Andrea Squatrito and I built Databoutique.com.

What’s inside

  • Why the Reddit API pricing change fits the shovels and gold rush analogy, and what it did to subreddit moderators.
  • Why every company defining its own data model leads to hundreds of scrapers hitting the same websites for roughly the same data.
  • What AI-based extraction services have shown so far on large e-commerce sites compared with smaller ones.
  • How a marketplace of standardized datasets changes the economics for sellers and buyers, and what it does to bot traffic on target sites.

In the full article: The Google Trends chart on web scraping interest, the links to our earlier tests of AI-based extraction on Amazon and Walmart, and the full Home Depot analogy for how Databoutique works.

Read the full article on The Web Scraping Club