Summary of an article published on The Web Scraping Club on 2025-10-26 (free to read). Read the full article
You test a scraper on a few pages, run it on the full site, and find fields missing because the page structures vary. Or the site changes, gets a new anti-bot layer, or disappears before you go back for the data you skipped. If you only kept the extracted fields, the raw page is gone and you have to scrape again, and pay for proxy traffic again.
I cover offline web scraping: download the pages first, extract the data later, at whatever speed and parallelism you want. I go through the cases where it makes sense, including a client project where I saved a whole site with millions of pages before it was shut down. I explain how the approach differs for static sites and for dynamic ones, where the rendered DOM matters more than the original HTML. I also cover how to get the pages (site downloaders, sitemaps, link-discovery crawlers), the open-source tools, and the two main challenges at scale.
What’s inside
- Online and offline scraping are not mutually exclusive: you can save the HTML while scraping live and parse it again later.
- Why storing the original HTML of a dynamic web app often does not work, and what to store instead.
- Practical tips on compression, S3 or database storage, per-page metadata and versioning of scraped pages.
- Why you may need to maintain two sets of scrapers, one for stored pages and one for the live site.
In the full article: The comparison table of offline and traditional scraping, the list of open-source website downloaders, and the detailed storage and versioning tips are only in the full article.