What it is
Contabo is a cloud provider that rents virtual private servers and dedicated servers.
How it works
Web crawlers have been identified making requests from Contabo IP address ranges. The analysis of these requests revealed a structure related to the Scraper Junkyard, a system where web pages are generated by a PHP program. This program assigns a “node number” to each page, and links between pages form a large binary tree structure.
The requests made by Contabo IP addresses against this system show a consistent structure based on these node numbers, which can be categorized into initial probes, irregular requests, and various series (B, C, D, E), along with sub-series.
TWSC experience
Not yet tested by TWSC.
Related
Sources
The Web Scraping Wiki is compiled from the tests and research published on The Web Scraping Club, the newsletter for web scraping professionals. Subscribe to get the next article in your inbox.