Summary of an article published on The Web Scraping Club on 2026-09-15 (free to read). Read the full article
AI agents need live web data, and most of us would rather not rebuild the scraping stack for every project. webclaw is a Rust toolkit, open source or managed cloud, that claims to be a drop-in Firecrawl alternative and to turn any URL into clean, LLM-ready content. It has over 2.3k GitHub stars, so it is worth a look before you commit to a tool.
I tested it hands-on, with a Scale plan with 1,000,000 credits provided by webclaw. I cover its architecture, where the same extraction engine sits behind the CLI, the REST API, the SDKs and the MCP server, and the trade-offs between self-hosting and the Cloud API. Then I ran it on Quotes to Scrape, its JavaScript version and an Open Food Facts product page. The examples cover output formats, JavaScript rendering, CSS selector filtering and BFS crawling, each through the CLI and the Cloud API.
What’s inside
- The self-hosted AGPL-3.0 build and the Cloud API differ on JavaScript rendering, proxy rotation, bot-protection handling and crawl limits.
- The MCP server is local-first and falls back to the cloud API when it hits a bot-protected or JavaScript-heavy page.
- Besides Markdown and JSON, there is an LLM-optimized output format meant to cut token usage.
- CLI crawls and Cloud API crawls behave differently: one is synchronous, the other asynchronous with a task ID.
In the full article: The CLI and curl commands for each example, the output screenshots, the Cloud API results on the JavaScript page, my note on anti-bot bypass, and my final take on webclaw.