What it is
Danubia is a clean text extraction API designed to take a URL as input and return the page’s main content formatted as clean markdown. It aims to provide a dependable data source by automating the process of creating clean datasets from web data.
How it works
Danubia utilizes its own content extraction library, which was rebuilt from scratch for speed and accuracy, drawing inspiration from state-of-the-art solutions like Trafilatura. This extraction process is the core value proposition, providing cleaner markdown output compared to raw HTML.
The underlying crawling infrastructure is designed to be polite, enforcing per-domain concurrency limits and crawl delays at scale. Furthermore, all users share a crawling infrastructure that includes a cache, allowing cached fetches to be served at a lower cost. A batch API is also available for scraping large datasets efficiently.
TWSC experience
Not yet tested by TWSC.
Sources
The Web Scraping Wiki is compiled from the tests and research published on The Web Scraping Club, the newsletter for web scraping professionals. Subscribe to get the next article in your inbox.