Summary of an article published on The Web Scraping Club on 2024-07-29 (for paid subscribers). Read the full article

Web scraping has always sat in a gray area. Look at LinkedIn and try to find a “Head of web data acquisition” at OpenAI, Perplexity or Mistral. They all have scraping teams, but few say so in public. I expected the AI boom to legitimize our industry. Instead, I hear more and more people talking about scraping for the wrong reasons.

I start from the basics: scraping is just a tool, like a hammer, and behavior makes it legal or not. I recap the legal points from Sanaea Daruwalla of Zyte and the ethical rules we follow, including server load, robots.txt and not interfering with the target’s business. Then I look at how AI companies behave against those rules. I cover the lawsuits over copyrighted material, user agents that were declared and then bypassed, and the complaints from site owners about bots pulling terabytes. I also look at what this means for sites that must buy bot mitigation, and for scrapers who stayed polite.

What’s inside

  • Which legal points we keep from Zyte’s guide: personal data, copyrighted material, and data behind logins and paywalls.
  • Why a declared user agent and robots.txt instructions mean little when bots ignore them, and why robots.txt is still a matter of netiquette.
  • The iFixit case: one million hits in 24 hours from a bot that was blocked in robots.txt, plus the bad engineering patterns site owners report.
  • How data deals, such as Reddit with OpenAI and Databoutique, try to give site owners revenue, and why no such solution exists for news and social sites.

In the full article: Our full list of ethical scraping rules, the links to the lawsuits, the Reuters report and the iFixit complaint, and the details of the monetization options for site owners.

Read the full article on The Web Scraping Club