Summary of an article published on The Web Scraping Club on 2023-09-20 (free to read). Read the full article

Web scraping is as old as the Internet, but most countries never gave it a proper place in their legislation. Still, it is used so widely that some high-level conclusions can be drawn. The 2023 wave of lawsuits over genAI training data made this hard to ignore, whether you collect web data or buy it. Hedge funds got there first, because in an SEC-regulated market material nonpublic information and insider trading are real risks.

We build the frame on two elements: the nature of the data and its visibility. For nature, we go through personal data, intellectual property and the sui generis database right, and where each one stops applying. For visibility, we separate non-public from public information, then look at terms and conditions and robots.txt. We are not lawyers, the rules change by geography and over time, and this is our own interpretation.

What’s inside

  • Which self-guidance bodies exist (IDSO and the FISD Alternative Data Council) and what they say about click-wrap versus browse-wrap terms of service.
  • Where the sui generis database right applies and where it does not, plus the exceptions based on the purpose and the extent of the scraping.
  • Two edge cases on what IP does not cover: airline prices and timetables before the European Court of Justice, and copyright on AI-generated content.
  • Why robots.txt matters even though it is a de facto standard from 1994 and not law.

In the full article: The links to the three 2023 lawsuits, the LinkedIn user agreement excerpt, the Farfetch robots.txt example and the reasoning behind each rule of thumb on personal data, IP and non-public access.

Read the full article on The Web Scraping Club