Summary of an article published on The Web Scraping Club on 2024-04-14 (free to read). Read the full article

When a scraper gets blocked, one of the first fixes we suggested in the course was changing its IP with a proxy. This episode of Web Scraping from 0 to Hero starts from the basics. The IP address itself matters little to an anti-bot. What matters is what can be derived from it: reputation built on past usage, geography, and the ISP or owner behind it. A cloud provider IP and a mobile carrier IP are treated very differently.

We define what a proxy is and sort proxies along two axes. The first is anonymity level: transparent, anonymous and high-anonymity (elite). The second is origin: datacenter, ISP, residential and mobile. For each type we cover what it is and how it fits scraping. For mobile proxies we also explain CGNAT, since it is the reason they are hard to block. We close with where to find proxies, from free lists to commercial vendors and directories.

What’s inside

  • How to read an IP’s ASN to spot a datacenter range, shown with a Google Cloud example
  • Which anonymity level matters most for scraping, and why transparent and anonymous proxies fall short
  • Why ISP proxies are described as a hybrid of residential and datacenter proxies
  • Why free proxy lists are fine for quick tests but not for recurring scraping jobs

In the full article: Screenshots of the ASN lookup, the full descriptions of each proxy type, the CGNAT explanation, and the links to proxy directories and provider maps.

Read the full article on The Web Scraping Club