Summary of an article published on The Web Scraping Club on 2025-06-15 (free to read). Read the full article
Anyone running a website knows scraping slows servers, raises bandwidth bills and exposes content. Blocking IPs and rate limiting no longer work well, because modern scrapers rotate IPs, change their fingerprint and try to act like human users. Even so, they leave small signals behind. Those signals are the triggers a detection system can learn from.
This time I look at scraping from the defender’s side. I sort the triggers into three groups: request patterns, session and behavioral signals, and headers and fingerprints. Then I walk through how supervised, unsupervised and semi-supervised learning can use them to tell scrapers from human traffic, with a sample training dataset and cluster plots. I also cover the hard part of feature work: pulling data from server logs, CDN logs, WAF logs and client-side JavaScript, and joining it into one dataset. The last section compares the three approaches on cost and weak points.
What’s inside
- The sample training dataset has ten behavioral features, from request rate and session duration to referer header, cookie acceptance and HTTP error rate.
- Server logs alone can miss a scraper that uses a headless browser and rotates IPs, because only client-side data shows the missing mouse movement.
- Feature building is split into four jobs: collecting data, making it uniform, linking related events across logs, and creating the features.
- The comparison covers labeling cost and concept drift for supervised models, false positives and explainability for unsupervised ones, and dependence on the first labels for semi-supervised ones.
In the full article: The sample training dataset table, the cluster and semi-supervised plots, and the full pros and cons of each ML approach for anti-bot detection are only in the full article.