Summary of an article published on The Web Scraping Club on 2024-12-08 (free to read). Read the full article
Proxy pricing scales with the size of a scraping operation, so at large volumes even a small percentage saved turns into real money. Proxies also come in very different types, from datacenter to ISP, residential, mobile and web unblockers. They run from the most detectable and cheapest to the most trusted and most expensive. Using an unblocker everywhere keeps bans away, but only works if you have infinite pockets.
I call my approach the Proxy Ladder: pick the cheapest solution that works, and climb to the next one only when it stops. We go through five steps, starting with no proxies at all and ending with web unblockers, using a simple Scrapy spider as the example. I also cover how mobile proxy pricing has changed over the past two years, and close with three tips for cutting the bill, including a way to avoid paying for datacenter proxies.
What’s inside
- What the errors tell you at each step: a rate-limited local IP, a blocked datacenter subnet, or an anti-bot problem that has nothing to do with IP reputation.
- Why mobile proxies are now a realistic option before web unblockers, and how their price moved compared with 2023.
- How the pricing model should follow the scraper: pay per GB or pay per request depending on whether it reads internal APIs or downloads HTML.
- The balance between using several providers for reliability and concentrating volume on one for discounts.
In the full article: Our walkthrough of each rung of the Proxy Ladder, the price comparisons between proxy types, and the three cost-cutting tips with the tool we suggest for building your own datacenter proxy network.