Summary of an article published on The Web Scraping Club on 2025-07-06 (free to read). Read the full article

Putting a proxy between your scraper and the target hides your IP, but the proxy IPs can be blacklisted too. Low-reputation datacenter ranges, a pool that is too small and bot-like behavior are the usual causes. Bans are not all the same: soft bans clear in seconds or minutes, hard bans are permanent, and network-wide bans take out a whole subnet. Buying proxies is not enough. You need to know what a ban looks like and what to do when it happens.

I go through the causes and the three ban types, then the HTTP status codes and error messages that may signal a proxy ban, with the caveat that some of them have other causes. For handling bans, I cover slowing down, retries with a delay, IP rotation and session hygiene. We then wrote a constant-backoff retry function for Python Requests and one for Axios in JavaScript. I close with the parameters that decide whether a retry setup works in production.

What’s inside

  • Rarer causes of proxy bans: geo-restrictions, manual bans by site admins and the bad neighbor effect on shared proxies.
  • Why a retry on a 407 proxy authentication error makes no sense, and why retries on a non-rotating proxy can just hit the same banned IP again.
  • Exponential backoff is the better choice for production. The examples use constant backoff to keep things simple.
  • The retry parameters to tune: strategy type, maximum retries, maximum delay and total retry time, and why they need adjusting as target sites change their anti-bot rules.

In the full article: Both retry functions (Python Requests and JavaScript Axios) with their usage examples, the table of error messages and status codes tied to proxy bans, and the table of example retry parameter combinations.

Read the full article on The Web Scraping Club