Summary of an article published on The Web Scraping Club on 2024-04-07 (free to read). Read the full article
After a webinar hosted by NetNut, I saw that many people still confuse a web unblocker with a traditional proxy. The confusion is understandable, since most unblockers are sold by proxy providers. In scraping, we define an unblocker as an API that lets our scraper bypass anti-bot protections through a set of dedicated features. That is a different job from changing the IP address the target sees.
I explain what an unblocker is and what features providers advertise. Since these are commercial products, we can only deduce the internals from provider websites, so I describe what we can reasonably assume happens behind a request. Then I set proxies against unblockers and say where one sits inside the other. I also cover whether any unblocker beats every anti-bot, with a pointer to my Great Web Unblocker Benchmark. Last, I weigh the cost of an unblocker against building your own bypass, and say how my own view has changed over the years.
What’s inside
- Why VPNs count as a broad kind of unblocker, but not in the scraping sense
- Why I assume requests first go through a plain proxy without a browser, and a browser session starts only when needed
- Why proxies are a component of an unblocker’s stack, and why proxy providers sell most unblockers
- Why I think one unblocker is rarely enough on a large scraping project, and how budget size shapes the build-or-buy decision
In the full article: The full feature list I extract from provider websites, my reasoning on how an unblocker routes requests, and my cost and time argument for buying versus reverse engineering anti-bots yourself.