What it is

Reader is an open-source, production-grade web scraping engine built for LLMs. It provides a simple API for scraping and crawling the entire web, producing clean markdown output. The project aims to handle the complexities of anti-bot defenses, TLS fingerprinting, proxy management, and reliability issues that arise in production.

How it works

Reader is built on top of Ulixee Hero, a headless browser designed for web scraping. It offers two main primitives: scrape and crawl. The scrape function allows you to scrape URLs and get clean markdown or HTML output, while the crawl function discovers and scrapes pages within a specified depth. Under the hood, Reader manages browser instances, anti-bot bypasses, proxy rotation, and other necessary tasks to ensure reliable and efficient scraping.

TWSC experience

Not yet tested by TWSC.

Sources


The Web Scraping Wiki is compiled from the tests and research published on The Web Scraping Club, the newsletter for web scraping professionals. Subscribe to get the next article in your inbox.