Summary of an article published on The Web Scraping Club on 2024-01-04 (free to read). Read the full article
The Lab is the series of The Web Scraping Club where we work on real issues on real websites: how to get past anti-bot X, how to make a bot indistinguishable from a human browsing. I started it because the only write-ups I found came from commercial vendors and ended with a pitch for their services. Some say talking openly about solutions only helps anti-bot companies fix their products. I think sharing does more good than harm, even if a solution stops working later.
Here I look back at the 35 Lab articles, written mostly in 2023, and at the three most read: the hRequests benchmark, the Anti-Detect Anti-Bot matrix and the early 2023 Cloudflare update. Targets ranged from OpenSea and Etherscan to Cloudflare, PerimeterX, Kasada and Akamai. Then I lay out 2024: The Lab goes from bi-weekly to weekly, and the matrix returns as a recurring issue.
What’s inside
- Why Lab articles and their code sit behind the paywall and in a private GitHub repository: they are the most time consuming to write and test.
- The 2023 run started with OpenSea and Etherscan, both behind Cloudflare at the time, and shows why solutions on real websites go stale.
- Fabien Vauchelles wrote a guest Lab on reverse-engineering a mobile app API with Charles Proxy and Android Studio.
- The Anti-Detect Anti-Bot matrix compared 5 solutions against 5 anti-bots, and the Cloudflare solution will probably need an update in 2024.
In the full article: The list of past Lab posts with links, the reasoning on keeping the code in a private repository, and the poll on what to cover next.