Summary of an article published on The Web Scraping Club on 2024-12-15 (free to read). Read the full article

In our scraping projects we find JSON in the responses to API calls and inside the HTML of the target websites. Many web frameworks use it to pass dynamic data from the backend to the front end. Extracting it is usually more reliable than parsing raw HTML. It is also cheaper, because with a pay-per-GB proxy plan the bandwidth saved shows up directly in the bill.

This is a beginner-level guide to handling JSON in Python with real-world examples. We start from what JSON is and why it maps so naturally to Python dictionaries. Then we go through the built-in json module: parsing strings, reading and writing files, pretty-printing, nested structures and editing values. After that we show how to read JSON responses in Scrapy and Playwright, and add a bonus on loading JSON into pandas. I also list the online tools I use to view and validate large JSON structures.

What’s inside

  • Seven reasons to prefer JSON over HTML as a scraping target, including proxy bandwidth cost and the stability of APIs compared to page markup.
  • Two online tools I use to inspect JSON: one that flags invalid JSON, for example when you copy several JSON objects from HTML at once, and one for saving and sharing.
  • The difference between the json functions for strings, files and output: loads, load, dump and dumps.
  • How the same parsing step looks inside a Scrapy spider and in a Playwright script, and how to load JSON into a pandas DataFrame.

In the full article: Code examples for every json function, the Scrapy spider, the Playwright script, the pandas snippet with its output, and the links to the JSON viewing tools.

Read the full article on The Web Scraping Club