Summary of an article published on The Web Scraping Club on 2023-02-07 (free to read). Read the full article

Web scraping is not illegal in itself, but whether a given project is legal depends on how we collect the data, which data it is and what we do with it afterwards. The limits come from several directions at once: website terms of service, intellectual property rights, privacy regulations and, in the U.S., the Computer Fraud and Abuse Act. A wrong call can end in a lawsuit or a cease-and-desist order.

I go through these four factors one by one and read them against two U.S. court cases, HiQ Labs vs. LinkedIn (2019) and Craigslist vs. 3Taps (2013). Then I list practical guidelines for staying compliant, covering terms of service, personal data, robots.txt and asking permission, and a short set of ethical practices that go beyond what the law requires, such as how we pace requests and when to prefer an API.

What’s inside

  • Why courts have not agreed on whether website terms of service can be enforced against scrapers, and when they treat them as too broad or ambiguous.
  • How GDPR in Europe and CCPA in California make extracting personal data without consent a legal problem.
  • What the CFAA means for publicly available data compared with data behind a password or an explicit restriction.
  • What happens when scraping continues after a site has explicitly banned the scraper.

In the full article: Details of each court case, the full compliance checklist and the list of ethical scraping practices, with the reasoning I give for each.

Read the full article on The Web Scraping Club