Summary of an article published on The Web Scraping Club on 2024-01-28 (free to read). Read the full article

I had a free trial of the Nimble professional plan, worth 3600 USD, and used it on production scrapers at Databoutique.com. Moving Scrapy spiders to Nimble residential proxies went smoothly. Then I picked up a backlog item: a scraper for a small Akamai-protected e-commerce site, with fewer than 1k items per country, that kept getting stuck in a JS challenge until max redirections.

I first tried the Nimble Browser, and the credits ran out before a single run completed. I then switched to Playwright with residential proxies, blocked images, and rebuilt the scraper around the site’s sitemaps. It worked on my laptop, but in production the account was emptied again with no decent execution. Using Nimble’s per-pipeline dashboard, I traced where the traffic went. We found a mix of planning mistakes and odd behavior from the website itself, and I turned them into three lessons on proxy budgeting.

What’s inside

  • Why grouping four websites under the same residential proxy credentials made the outlier hard to spot.
  • How much traffic third-party services called by the target page consumed compared with the target itself.
  • Why blocking only images in Playwright was not enough, and why the same fix does not apply to Scrapy spiders.
  • A page-loading loop seen only from a datacenter machine, and why a usage threshold matters even on a trivial scraper.

In the full article: Screenshots of the Nimble pipeline and usage dashboards, the Playwright route interception code, the full breakdown of traffic per domain and the three lessons in detail.

Read the full article on The Web Scraping Club