Summary of an article published on The Web Scraping Club on 2023-01-22 (free to read). Read the full article

Open-source code does a lot of the work in web scraping, and the useful repositories are scattered across languages and use cases. Finding the ones worth your time means sorting through frameworks, browser automation tools, proxy helpers and anti-bot workarounds, and knowing which ones are maintained and by whom.

We put together a list of the repositories we find most interesting, grouped by language: Python, JavaScript and other languages such as Go and Ruby. Within each group we cover scraping frameworks, proxy management, tools aimed at Cloudflare and TLS fingerprinting, and a few more unusual projects. A final section collects knowledge bases and documentation, including an “awesome” list that is much more complete than ours. Some entries are tools I use, one is a package I wrote myself, and one I had never seen before and plan to test.

What’s inside

  • Which Python projects sit around Scrapy: Splash for JavaScript rendering and Spidermon for scheduling and monitoring a fleet of scrapers.
  • How the proxy packages differ, including one that borrows the AWS API Gateway IP pool as a cheap alternative to data center proxies.
  • Which tools target anti-bot systems: Cloudscraper and TLS-client in Python, puppeteer-extra and playwright-extra in JavaScript.
  • Less common picks: Autoscraper learning extraction rules from examples, Ayakashi using an SQL-like language on the DOM, and a journalism scripts collection from Stanford.

In the full article: Repository links, the star count for Scrapy, the one-line verdict on each of the roughly twenty projects, and the full list of knowledge bases and documentation sources.

Read the full article on The Web Scraping Club