Summary of an article published on The Web Scraping Club on 2023-09-02 (free to read). Read the full article

August’s biggest news was that The Web Scraping Club turned 1 year old, so this recap is not a list of news. After ten years in web scraping, I still find useful content scattered around: marketing material for services, a few hidden blogs by great hackers that are hard to follow, and little practical material in one place. I started taking notes for myself and, last year, began sharing them.

I describe the two kinds of posts, Educational and Solutions, and why you won’t find antibot reverse engineering write-ups here. Customers wait days, not months, and such a post is stale after the next update. I also explain how I plan to split free and paywalled content over the next 12 months, and I ask which of our recurring formats you like most.

What’s inside

  • Why a good scraping professional needs to understand why blocks happen and also know the right tools to fix them in a timely manner.
  • Which series stay free, like Hands On where we test commercial tools, and which stay paywalled, like The Lab with its code repository.
  • Where the community meets (Discord, Twitter, Reddit) and why I also publish on a WordPress blog, given Substack’s weak SEO.
  • What I want to try in the coming months, from longer-form content to different mediums.

In the full article: The polls on preferred formats, the links to the community channels, my reasoning behind the free and paywalled split, and the plans for longer-form content and new mediums.

Read the full article on The Web Scraping Club