Summary of an article published on The Web Scraping Club on 2025-01-26 (free to read). Read the full article

AI agent became the buzzword after LLM. An agent takes a prompt, splits it into subtasks, retrieves data from several websites and composes an answer. For us the interesting step is data retrieval: the agent needs a browser, has to automate it and has to read the rendered DOM, including dynamic content.

Browser automation sounds like the whole answer, but anyone who has deployed a fleet of Playwright instances knows it is neither easy nor cheap. I start from my own case, scraping full e-commerce catalogs country by country, and compare it with Scrapy on a small VM. Then we go through the stealthier alternatives, web unblockers and anti-detect browsers, and what each one gives up. Finally we look at the new services that host browser sessions for agents, including LightPanda, a browser built for machines rather than humans, and at how the needs of agents overlap with ours.

What’s inside

  • Why Playwright is my last resort and what it costs in VM size and concurrency compared with Scrapy
  • What web unblockers do well for browserless scrapers and the one thing you cannot do with them
  • Why anti-detect browsers still share the hardware limit of every browser automation solution
  • Which services run Playwright and Puppeteer sessions over WebSocket, and the one real difference between agent tasks and scraping runs

In the full article: Details of my per-country VM setup on AWS, the full comparison of unblockers and anti-detect browsers, the list of browser infrastructure services and the plan for the upcoming reviews.

Read the full article on The Web Scraping Club