Summary of an article published on The Web Scraping Club on 2023-06-04 (free to read). Read the full article

AI Month on The Web Scraping Club opens with a question every practitioner has: where does AI actually help in scraping? A good place to start is a vendor that has built its product around it. Nimble sells an AI-driven scraping browser, and its founder, Uriel Knorovich, came to web data from cyber intelligence rather than from the scraping world.

I interviewed Uriel for our monthly interview, in collaboration with Nimble. We talked about why a team with a cyber background built a browser from scratch instead of patching Puppeteer and Selenium, and why they started applying AI and LLMs to it early. I asked whether AI is used for the anti-bot phase, for HTML parsing, or for both. I also raised my main concern about AI in scraping: a black box you cannot fix when a selector goes wrong. We then moved to where the industry is going, and ended with a story from his school days.

What’s inside

  • Why the team built its own browser, and how investors reacted to the idea.
  • How Nimble says it uses AI in the fingerprint engine and in proxy management, as well as in parsing.
  • Uriel’s answer on transparency, manual fine-tuning and a feedback loop for users.
  • His view on whether AI will replace scraper programmers, and on the gap between AI demos and cost-efficient real-world use.

In the full article: Uriel’s complete answers to all ten questions, including his account of the early days of Nimble and the school scraping story that ends the interview.

Read the full article on The Web Scraping Club