What it is
Kimurai is a Ruby-based web scraping framework that uses AI to assist in data extraction. It allows users to write scrapers using a clean, AI-assisted Domain-Specific Language (DSL). Kimurai uses AI to generate selectors on the first request and caches them for subsequent requests, enabling fast and efficient data extraction without the need for per-request AI calls.
How it works
- On the first request,
extractsends the HTML and your schema to an LLM, which generates XPath selectors and caches them in a file. - All subsequent requests use cached XPath selectors — zero AI calls, pure fast Ruby extraction.
- Kimurai supports OpenAI, Anthropic, Gemini, or local LLMs via Nukitori.
Kimurai also works as a traditional scraper, supporting headless antidetect Chromium, Firefox, or simple HTTP requests.
TWSC experience
Not yet tested by TWSC.
Related
Sources
The Web Scraping Wiki is compiled from the tests and research published on The Web Scraping Club, the newsletter for web scraping professionals. Subscribe to get the next article in your inbox.