Summary of an article published on The Web Scraping Club on 2023-11-05 (free to read). Read the full article
Most scrapers that need rework were started too early. Web scraping is no longer plain vanilla: anti-bots are everywhere, there are hundreds of tools, and picking the wrong one costs time and money. So before writing any code, and assuming we are allowed to scrape the site, a short round of checks pays off.
This is the second lesson of the WSF0TH course. I go through the checks in the order I run them. First we define the output data model and locate each field on the site. Then we look for the cleanest source: an API behind the website, an API used by a mobile app, or JSON embedded in the HTML. If none of those exist, we decide between plain HTML and JavaScript rendering, which drives the choice of tool. Then we look at the site’s architecture: which anti-bot it runs, what robots.txt says, and whether the sitemap can do the crawling for us.
What’s inside
- Why the output data model comes first, and what rework looks like when a field turns out to sit on another page.
- The advantages of APIs over HTML parsing, including data the website never shows, such as inventory levels.
- How the HTML versus JavaScript-rendered question maps to Scrapy or to browser automation like Selenium, Playwright or Puppeteer, and what it means for operating costs.
- Which free extension we use to spot anti-bots, and why a quick look at robots.txt can occasionally give a quick win.
In the full article: The screenshots of the network tab and of the Wappalyzer reports for Cloudflare and Kasada, the links to the earlier Fiddler and Charles Proxy lessons on app APIs, and the full reasoning behind each check.