Summary of an article published on The Web Scraping Club on 2023-07-30 (free to read). Read the full article
LLMs changed the economics of web scraping in 2023. Models like GPT and Bard train on scraped data, while sources such as Stack Overflow lose traffic to the same tools. Platforms like Twitter and Reddit, sitting on a mountain of data, respond by pushing paid APIs and tightening against scrapers. Anti-bot use goes up, and so does the cost of getting the data.
In this July recap I look at that tension and compare it with what happened to Google News in Spain. I then cover Google’s Web-Environment-Integrity proposal, which would let websites request an attestation of the browser and the hardware it runs on, and what it could mean for scrapers and for real users on less common platforms. The recap also covers Databoutique, our marketplace for selling scraped data, and the dates for the Extract Summit 2023.
What’s inside
- Why fewer Stack Overflow visitors make LLM training data less accurate and less timely, and put the source itself at risk
- The Google News shutdown in Spain as a precedent, and why I think the AI case is worse
- Who could act as attester for Android, Windows and Apple devices, and the open question of the fragmented Linux world
- How to start selling data on Databoutique, and how to request a website that is not yet listed
In the full article: Links to the sources behind each topic, the full text of the Web-Environment-Integrity proposal quoted from its GitHub repository, and the Extract Summit dates and ticket details.