Summary of an article published on The Web Scraping Club on 2022-12-30 (free to read). Read the full article
In 2022 anti-bot solutions gave us the most headaches. Scraping got harder, and doing it at scale got more expensive. It did not slow the industry down. It pushed developers toward more sophisticated techniques, more proxies and headful browsers, while spending on web data kept growing, as Shane Evans of Zyte said at the 2022 Extract Summit.
I look back at the year from where I work, at Re Analytics, scraping hundreds of e-commerce websites every day since 2014. I pick three trends: web scraping getting legal recognition through court rulings in the US and Korea, the arrival of AI after I was skeptical for years, and no-code tools that improved enough to change my mind in some cases. I also explain why I started The Web Scraping Club, how it grew in its first months, which post performed best, and what I want to change in 2023.
What’s inside
- Which court decisions shaped legal and illegal scraping in 2022, and where the line stays, such as personal data gathered without consent.
- Why ChatGPT changed my view on AI for scraping, and what would have to happen for it to become practical.
- The kind of target where no-code tools are now the fastest route to value, and the kind where I still doubt them.
- Why the Club exists, the Discord community, the topics I want new contributors to cover in 2023, and what happened when I shared posts on Hacker News.
In the full article: The links to the court cases and to the Zyte talk, my full reasoning on AI and no-code tools, the Club’s growth figures, and the plans for 2023 are in the full article.