Summary of an article published on The Web Scraping Club on 2022-10-02 (free to read). Read the full article

Anyone who scrapes large sites knows that too many requests from one IP get you blocked. That is why we buy proxies. What I did not know is how far the block can reach on targets like Google or Amazon, which are hit hard by bots. Picking a provider on price alone can cost you more than it saves.

I start from the basics of what an IP address is and how ISPs get their ranges from the regional Internet registries. Then I go through the two aspects Neil from Rayobyte, a vendor of data center, residential and mobile proxies, says every proxy user should check: diversity of IPs across subnets, and IP reputation. For each I explain why it matters for a scraping project, using a worked example with Amazon-style request limits for the first. I close with a short fun fact on IPv4 prices.

What’s inside

  • How a block on one IP can extend to the other addresses in its subnet, and what the “noisy neighbor problem” means for your request budget.
  • Why diversity of sources in the IP rotation matters for scrapers and for the proxy providers themselves.
  • How anti-bot software uses blacklists and IP reputation, with the case of the IP addresses stolen from AFRINIC.
  • Why proxy prices that look too good to be true often point to poor IP reputation.

In the full article: The worked subnet example with request counts, the AFRINIC story, the IPv4 and IPv6 background and the note on IPv4 prices are only in the full article.

Read the full article on The Web Scraping Club