Summary of an article published on The Web Scraping Club on 2023-02-17 (free to read). Read the full article
The web works like an enormous library, but being able to read a page doesn’t mean we can scrape it. A scraper can behave perfectly on the technical side and still create legal problems, because public accessibility and permission are two different things. This matters for anyone who builds data products on scraped content and assumes that public means free to take.
Here I follow up on my earlier piece on whether web scraping is legal and add the rules that come after it. I cover personal data and privacy law, copyright on images and text, and the case of scraping a website to get an advantage against it. I then put everything into a short list of rules for ethical web scraping, with a video that goes into more detail.
What’s inside
- Which privacy rules apply depends on the nationality of the person whose data we scrape, so there is no single set of rules to follow.
- Flickr is the copyright example: the photographers own the rights, not us, so we can’t publish their pictures, let alone resell them.
- Filling a competing listing website with data scraped from another one is the example of scraping against the target.
- The ethical list also covers not flooding the target with requests and not scraping behind a login, since a ToS is explicitly accepted there.
In the full article: Cross-references to my earlier pieces on scraping legality and on social networks, the exact wording of the four ethical rules, and the embedded video on ethical web scraping.