Summary of an article published on The Web Scraping Club on 2024-03-31 (free to read). Read the full article

Sooner or later every scraping professional has to answer “Why is my scraper getting blocked?“. The cause is rarely obvious, and an analysis you made when you wrote the scraper may no longer hold months later. Without a method you end up changing proxies, headers and browsers at random.

This lesson of the “Web Scraping from 0 to Hero” course is the playbook I use daily when a block hits one of my scrapers. It is a decision path. We start by checking whether the target runs an anti-bot, using Wappalyzer. Then we see where the scraper runs: on a local machine, on a datacenter, or on neither. At each step we separate browserless scrapers from those using browser automation. The last branch covers scrapers that run for a while and then get blocked. For each anti-bot I point to the collection of articles already published.

What’s inside

  • Why an anti-bot check done months ago may be outdated, and when a browser automation scraper can be rewritten as browserless.
  • What changes when the scraper works on your machine but fails on a datacenter, and why cloud providers like AWS make this easier for websites to detect.
  • What to check when the scraper fails even locally, separately for browserless scrapers and for browser automation ones.
  • The causes of a block that appears mid-run: session rate limits, IP pools that are too small, subnet bans, and overloading a small website.

In the full article: The full decision path with every branch, the Wappalyzer step, the header comparison screenshot and the links to our guides for Akamai, Cloudflare, Datadome, Kasada, PerimeterX and Shape/F5.

Read the full article on The Web Scraping Club