Summary of an article published on The Web Scraping Club on 2023-06-04 (free to read). Read the full article
AI Month on The Web Scraping Club opens with a question every practitioner has: where does AI actually help in scraping? A good place to start is a vendor that has built its product around it. Nimble sells an AI-driven scraping browser, and its founder, Uriel Knorovich, came to web data from cyber intelligence rather than from the scraping world.
I interviewed Uriel for our monthly interview, in collaboration with Nimble. We talked about why a team with a cyber background built a browser from scratch instead of patching Puppeteer and Selenium, and why they started applying AI and LLMs to it early. I asked whether AI is used for the anti-bot phase, for HTML parsing, or for both. I also raised my main concern about AI in scraping: a black box you cannot fix when a selector goes wrong. We then moved to where the industry is going, and ended with a story from his school days.
What’s inside
- Why the team built its own browser, and how investors reacted to the idea.
- How Nimble says it uses AI in the fingerprint engine and in proxy management, as well as in parsing.
- Uriel’s answer on transparency, manual fine-tuning and a feedback loop for users.
- His view on whether AI will replace scraper programmers, and on the gap between AI demos and cost-efficient real-world use.
In the full article: Uriel’s complete answers to all ten questions, including his account of the early days of Nimble and the school scraping story that ends the interview.