Summary of an article published on The Web Scraping Club on 2024-03-19 (free to read). Read the full article

Web unblockers are commercial “super APIs” that handle JS rendering, captcha solving, IP rotation and fingerprint forging, so your spider doesn’t break when a target changes its anti-bot. They cost more than plain proxies, billed per GB or per request. They also save you the hours spent fixing code and the weight of running browser automation. Before picking one, you want to know which anti-bots each provider actually beats, how fast, and at what price.

This is the first edition of a new series, The Great Web Unblocker Benchmark. We tested Bright Data, Oxylabs, Smartproxy, Zenrows and Zyte API with the same simple Scrapy spider, capped at 10 concurrent connections, against Cloudflare, Datadome, Kasada and PerimeterX on live e-commerce sites. We measured success rate, execution time and extraction cost. The vendors did not see the results beforehand, and this is not a paid review. I also explain the limits of the setup and how the pricing models differ.

What’s inside

  • Pricing models compared: pay-per-GB for Smartproxy and Oxylabs, pay-per-request for Bright Data and Zyte, credits for Zenrows, and why per-GB gets costly on pure HTML.
  • Test flow: home page, a product category, then the first 20 items. Datadome is the exception, where the product pages are requested directly.
  • Caveat on the method: a solution that fails here may still work inside a Playwright scraper, because some anti-bots look at mouse movements and clicks.
  • Next in the series: a second unblocker round in June, and an anti-detect browser benchmark in April.

In the full article: The per-provider results for each anti-bot, the timing and cost recap tables, the exact targets and the notes on retry policies are only in the full article.

Read the full article on The Web Scraping Club