Summary of an article published on The Web Scraping Club on 2025-05-22 (for paid subscribers). Read the full article
LLMs have changed how scrapers get built, but “AI scraping” now means very different things: a coding assistant, an AI-enhanced IDE, or a hosted API where you pass a URL and the output structure you want. Which one makes sense depends on how many sites you target, how well they are protected, and what a page costs you compared with writing the scraper yourself.
We compared five approaches: OpenAI Codex, Cursor with Model Context Protocol and rules, ScrapeGraphAI, Firecrawl and Zyte API with AI extraction. For Codex, still in beta and tied to a GitHub repository, I share my first impressions. For Cursor, we look at how custom tools and rules shape the code it writes. For the API tools, we look at what they handle for you and where LLM-based extraction stops working. The ScrapeGraphAI test code comes from my PragueCrawl talk.
What’s inside
- What Codex’s sandbox and fixed context window mean when you try to feed it a real page, and why prompting has to tell it which parts of the HTML matter.
- How Cursor rules (XPath only, scraper templates) cut down the fuzziness of generated code, and where MCP tools stop being enough, such as internal APIs, forms and CAPTCHAs.
- The cost reasoning for API-based extraction: 1 million pages from one site versus 1 million pages from 10k sites, set against the time and cost of building 10k scrapers.
- ScrapeGraphAI open source versus its API, including the anti-bot bypass under the hood and the data no LLM-powered tool can reach when it sits behind a hidden website API.
In the full article: The scripts from the 84.AI-WEB-DRIVEN-SCRAPING repository, my verdict on each tool, the ScrapeGraphAI API extraction code, and the Firecrawl and Zyte API sections.