Summary of an article published on The Web Scraping Club on 2023-04-30 (free to read). Read the full article

April 2023 was dominated by AI. After my earlier piece on GPT4 and its consequences for the industry, other people started building on the same idea, and new projects keep appearing. The practical question for anyone running scrapers at scale is whether a tool that takes a data schema and a URL and returns parsed data can replace selectors and the maintenance they need. A second topic was a FastCompany article accusing big tech of double standards on scraping.

In this monthly recap I look at Scrapeghost and Kadoa and say where I have doubts: cost efficiency, anti-bot bypass and the black-box nature of the output. I also say how I would combine them with unblocking products such as the new Zyte API or Bright Data Web Unlocker. Then I give my view on the FastCompany piece, where I agree with part of the journalist’s thesis and disagree with the rest. The recap also covers the video of the month, the most-read article of April and a subscriber milestone.

What’s inside

  • Cost is judged in two ways: execution costs plus the AI service, and the human hours saved on fixing broken spiders.
  • Why GPT-based tools cannot handle anti-bot bypass, and why that step sits earlier in the web data acquisition pipeline.
  • Why a black box that returns the data leaves no middle way when results are wrong: you either use it or you don’t.
  • My position on scraping Facebook through the Enterprise API versus scraping it directly, and the case for a free public API on political advertising.

In the full article: The links to Scrapeghost, Kadoa, the FastCompany article, the video of the month and the most-read post, plus my full reasoning on each point.

Read the full article on The Web Scraping Club