Summary of an article published on The Web Scraping Club on 2025-11-23 (free to read). Read the full article

Scraping gives you data, but stakeholders want structure. Unless you pull a single table, the logic does not come for free. Someone asks you to sort thousands of scraped images into folders by topic, or to say what a pile of reviews is about, and doing it by hand is slow or impossible.

I start by clarifying what “using AI” means: machine learning, deep learning or generative models, with no need for a fully autonomous agent. Then I split pattern detection into a classical method and a modern one. The classical one is unsupervised learning, and I treat structured and unstructured data separately, with Python examples built on scikit-learn and sentence-transformers. The modern one is LLMs used as pattern finders on scraped data, including multimodal inputs like images, PDFs and audio. In both cases I look at what you have to tune and where it stops working.

What’s inside

  • K-means on numerical data such as scraped real estate listings, and why you must choose the number of clusters yourself
  • Clustering review text through sentence embeddings, with t-SNE to plot the result in two dimensions
  • Topic modeling on scraped news headlines with LDA, and why a single run can return a blended, unclear topic
  • LLM use cases such as thematic grouping of reviews and anomaly detection in server logs, and why it is not a coffee-sipping job

In the full article: Python code for k-means, review embeddings and LDA topic modeling, the plots and topic outputs, and the full list of LLM prompts and use cases.

Read the full article on The Web Scraping Club