Summary of an article published on The Web Scraping Club on 2023-09-02 (free to read). Read the full article
August’s biggest news was that The Web Scraping Club turned 1 year old, so this recap is not a list of news. After ten years in web scraping, I still find useful content scattered around: marketing material for services, a few hidden blogs by great hackers that are hard to follow, and little practical material in one place. I started taking notes for myself and, last year, began sharing them.
I describe the two kinds of posts, Educational and Solutions, and why you won’t find antibot reverse engineering write-ups here. Customers wait days, not months, and such a post is stale after the next update. I also explain how I plan to split free and paywalled content over the next 12 months, and I ask which of our recurring formats you like most.
What’s inside
- Why a good scraping professional needs to understand why blocks happen and also know the right tools to fix them in a timely manner.
- Which series stay free, like Hands On where we test commercial tools, and which stay paywalled, like The Lab with its code repository.
- Where the community meets (Discord, Twitter, Reddit) and why I also publish on a WordPress blog, given Substack’s weak SEO.
- What I want to try in the coming months, from longer-form content to different mediums.
In the full article: The polls on preferred formats, the links to the community channels, my reasoning behind the free and paywalled split, and the plans for longer-form content and new mediums.