Summary of an article published on The Web Scraping Club on 2024-11-10 (free to read). Read the full article

AI did not start with LLMs. Dynamic pricing, clustering, supervised classification and sentiment analysis have been in commercial use for about twenty years, and almost any company can build them. ChatGPT made generative AI something everyone could touch, and that changed the demand for web data I see every day.

I’m not an AI specialist, so I don’t go into technical details. I compare traditional and generative AI on two points: scope, one task versus an open request, and scale, which sets how much data you have to collect and who can afford to train a model. Then I look at how generative AI affects scraping: data management, the quality of structured versus unstructured data, how data is collected, and copyright. I close with what this pressure means for the scraping industry and for the websites on the other side.

What’s inside

  • Why a traditional AI project has a limited scraping scope, such as prices or social media, while an LLM needs a much wider and less predictable range of sources.
  • Why most companies should customize an existing model instead of training an LLM from scratch.
  • Why checking quality is easier on prices and factual fields than on text from social and news feeds, and what that means for filtering harmful content.
  • How copyright and traffic loss for publishers connect to the Google News case in Spain.

In the full article: The comparison with the Google News case in Spain, the ChatGPT knowledge cutoff example, the screenshots on scraping practices, and my view on how anti-bots and paywalls will respond.

Read the full article on The Web Scraping Club