Summary of an article published on The Web Scraping Club on 2023-03-09 (free to read). Read the full article

Bright Data Web Unblocker is an API sold as a way to scrape even the most protected websites. It plugs into a scraper like a normal proxy, and the vendor lists what happens behind it: per-IP rate limits, user emulation, device imitation, referrer calibration, honeypot detection and random delays between requests. That is easy to claim. What matters is whether it gets a plain HTTP scraper past real anti-bot systems without Playwright or Selenium.

We built a plain Scrapy spider that fetches 10 product pages from each of five retail sites, one per anti-bot: Datadome, Cloudflare, Kasada, F5 and PerimeterX. The spider without a proxy is the benchmark. We then ran it through Web Unblocker from a local machine and from a cloud VM, to see whether a server fingerprint leaks, and scored the outcome from 0 to 100.

What’s inside

  • Which site stands in for each anti-bot, including a Cloudflare target with one of the strictest policies I’ve seen.
  • How the score is built: a page counts only if the product string is found, across a local run and a server run.
  • Why we repeated the test from a cloud VM instead of trusting the local result.
  • The pricing caveat: per-GB billing versus billing per 1,000 requests, and what it means for large or low-budget projects.

In the full article: Result tables for the no-proxy, local and cloud runs, the proxy zone setup in Bright Data, the link to the test spider on GitHub, the final score and my pros and cons verdict.

Read the full article on The Web Scraping Club