Summary of an article published on The Web Scraping Club on 2024-09-01 (free to read). Read the full article

AI needs more and more data, so there is more web scraping. As we’ve seen especially in the past five years, there are also more anti-bots. Picking tools that help on both fronts matters, and open source keeps the cost down.

I put together my shortlist of Python open-source libraries for the two jobs: using AI in scraping and getting past anti-bots. On the AI side there is ScrapeGraphAI, and I say where I think LLMs fit in a scraping project and where they don’t yet. On the anti-bot and infrastructure side I cover Scrapoxy, Botasaurus, Nodriver, Undetected Playwright and Camoufox. Some of them I tested in earlier articles. Others I haven’t tested yet, and I say so for each one. I also ask for the packages I missed.

What’s inside

  • Which scraping tasks I think LLMs suit best, given that their response time is still too slow for production speed.
  • How Scrapoxy handles datacenter proxies by rotating virtual machines on different cloud providers, and how it exposes mixed proxy types through a single endpoint.
  • The limit I found in Botasaurus when running a headful scraper from a datacenter.
  • How Camoufox is built on a stripped-down Firefox with TLS masking and Browserforge, and why I want to test it.

In the full article: Links to the GitHub repositories and to my earlier tests of ScrapeGraphAI, Scrapoxy, Botasaurus and Undetected Playwright, plus the Rebrowser Blog reference on CDP detection and the alternative projects I mention.

Read the full article on The Web Scraping Club