Summary of an article published on The Web Scraping Club on 2025-05-11 (free to read). Read the full article
Training an unbiased AI model takes historically valid data, not only fresh data. Search engines do not help here, because SEO rules push sites to hide, archive or delete old pages. The Wayback Machine keeps those pages, with hundreds of thousands of snapshots for some of them, and old snapshots usually predate CAPTCHAs, IP bans and JavaScript-heavy rendering.
I use it as a scraping target through a concrete case: ESPN’s 2004 NBA Draft Top 10s by position, a page that no longer exists on the live site. We scrape it with Requests and Beautiful Soup, with no browser automation. The page is built on plain HTML tables, so we extract by column position and use a regex to split data crammed into a single cell. The data could then be compared with the players actually drafted. I also cover why the parsing logic depends on the snapshot date, since layouts change between versions, and what to do when you scrape the same page across several years.
What’s inside
- How a Wayback Machine URL is built from the snapshot timestamp and the original page URL.
- Why 2004 pages need only an HTTP client and an HTML parser, and what that means for old table-based layouts.
- Why one parser may not work across snapshots of the same page, and the option of an AI-powered parser to avoid rewriting it for each version.
- A reminder to keep request volume low, since the Wayback Machine is a nonprofit project.
In the full article: The full Python script, the CSS selector and regex used on the ESPN tables, the sample JSON output and the link to the GitHub repository folder.