Summary of an article published on The Web Scraping Club on 2026-02-22 (free to read). Read the full article
FireCrawl’s /scrape endpoint, Bright Data’s Web Unlocker API and Crawl4AI all offer Markdown output, and sometimes it is the default. If you build or buy a scraper that feeds an LLM, the format decides your token bill and how clean the model’s input is. Plain HTML carries a lot of markup the model does not need.
I cover why Markdown suits LLMs: structure, size, and its weight in training data. I compare the same ESPN page in HTML and Markdown, in bytes and in tokens. Then I show why a direct HTML-to-Markdown conversion is not enough: script and style blocks, ads, navigation, and boilerplate text all leak into the output. I wrote a short Python script with Beautiful Soup and markdownify that handles the first cases and measured it on a real page. I also look at where Markdown loses information and raw HTML is the better input.
What’s inside
- Four noise sources that survive a naive conversion: non-content tags, ads, navigation with headers and footers, and repeated boilerplate such as cookie and GDPR notices.
- Why boilerplate is the hard case, and what some industry leaders do about it with small LLMs trained for the filtering step.
- Why we copy the rendered outerHTML from the browser and not the page source, and what that changes for dynamic content.
- Cases where HTML beats Markdown: data-* attributes, ARIA attributes, class names and comments, with an Amazon product page example.
In the full article: The complete Python script, the step-by-step usage, the byte and token counts for each comparison, and the Amazon data-state example are only in the full article.