Summary of an article published on The Web Scraping Club on 2023-08-10 (free to read). Read the full article

Almost every month this year a new unblocker shows up, and choosing the right one for your case is getting harder. The feature list is always the same: proxy-style integration, rotating IPs, session handling and Javascript rendering. What matters in practice is price and how well it holds against the anti-bots that block real targets.

We tested the Oxylabs Web Unblocker in this Hands On episode, with the same method we used on other unblockers. A plain Scrapy spider requests 10 pages from each of 5 websites, one per anti-bot: Datadome, Cloudflare, Kasada, F5 and PerimeterX. We score from 0 to 100 on two runs, one local and one from a server, and the base scraper sets the benchmark at 0. We ran it without the unblocker, with raw HTML only, and with Javascript rendering. I also went through account setup and the usage dashboard.

What’s inside

  • Each test URL is checked with an XPath-populated field, so a 200 status alone does not count as a success when a challenge page comes back.
  • We compare raw HTML requests with Javascript rendering, to see what rendering adds when the target already lets the request through.
  • The usage dashboard splits traffic by domain, which helps when one proxy URL serves several sites. I say what I would add to it.
  • The closing section lists pros and cons, covers the pricing model per GB and the free trial, and ends with a final rating.

In the full article: Result tables for each of the three runs with the per-anti-bot breakdown, the link to the test spider on GitHub, the request setup for rendering, the pricing figures and the final score.

Read the full article on The Web Scraping Club