Summary of an article published on The Web Scraping Club on 2023-06-08 (free to read). Read the full article

I needed a new TV and found that Google Shopping and price aggregators could not be trusted. They list only retailers that send feeds, their data lags behind prices that change two or three times a day, and similar models get mixed together. Writing my own scrapers helped only in part: one scraper per site, Cloudflare on some targets, Amazon banning fast IPs, and weeks of maintenance. Any scraping project carries the same costs.

We tested Nimble Browser, a web driver built by Nimble and reachable via API or CDP, with embedded proxy management, fingerprint masking and optional AI parsing of the page. The target was one Sony TV model on Amazon, mediaworld.it, comet.it, unieuro.it and e-stayon.com, sites with different layouts and different anti-bot protections. We sent one request per site and looked at what the AI parsing returned on each.

What’s inside

  • Why I stayed skeptical of AI parsers: a black box is hard to adjust, and getting past the anti-bot is harder than parsing the HTML.
  • How Nimble Browser runs on its own infrastructure, and what that means for the machines you would otherwise size for headful Playwright.
  • What the JSON output contains besides the parsed data, and why that matters when the parsing misses something.
  • My verdict, with pros and cons, on pricing and on the kind of project it fits.

In the full article: Python request code, the headers used, the parsed output for each of the five sites, the screenshots and the GitHub repository with the full scripts.

Read the full article on The Web Scraping Club