Summary of an article published on The Web Scraping Club on 2024-09-08 (free to read). Read the full article

Not a day goes by without a new AI scraping tool being announced. In my whole career in web scraping I have never seen anything like it. Startups in the field are getting into YCombinator for the first time, and open-source projects, no-code tools and established vendors are all racing in the same space. For anyone who has to pick a tool, it is hard to tell what is actually what.

I tried to put some order in it by sorting the tools along two axes. The first is whether they use public models, typically LLMs like GPT, or AI developed in-house. The second is whether the models run on your machine or the work happens in the cloud. That gives four quadrants, and I placed every tool I could find that claims AI on its website, from Nimble, Zyte API and Kadoa to Octoparse, n8n and ScrapeGraph-AI. The map is not exhaustive, and I rely on what vendors say on their sites.

What’s inside

  • Which tools sit in each of the four quadrants, from private cloud APIs to self-hosted open-source projects.
  • Why Bardeen, Make.com and n8n are closer to automation frameworks with connectors than to scrapers.
  • Where Octoparse, AnyPicker and ScrapeStorm land, the ones that need a client or browser extension.
  • A caveat on vendor claims: I can’t verify the AI inside commercial engines, so the map follows their websites.

In the full article: The full map graphic and the complete tool list for each of the four quadrants, with the short description I wrote for every entry.

Read the full article on The Web Scraping Club