Summary of an article published on The Web Scraping Club on 2023-10-22 (free to read). Read the full article

Questions on our Discord server and on the webscraping subreddit keep showing the same gap: people lack the basics. Courses on machine learning and AI are everywhere, while web scraping material is mostly marketing that ends with “buy a commercial solution”, or sits in hackers’ forums. Most people also focus on one tool or one issue and miss the big picture. Being a professional means more than solving a technical challenge. You also need to know whether what you are doing is legal and whether a wiser way to extract the data exists.

That is why I started a free course, tagged WSF0TH, with a bi-weekly schedule. This first chapter defines web scraping and separates it from screen scraping and web crawling. It then looks at why demand is growing, with e-commerce growth, market intelligence and AI training data as drivers, and at the freelance and data marketplace options for earning from it. The last part covers legality: which types of data we can collect and how accessible they are.

What’s inside

  • How web scraping differs from screen scraping and web crawling, with a short definition of each.
  • Why the Google search trend and the growth in e-commerce sites since COVID-19 point to rising demand.
  • Where practitioners can earn from scraping today: Upwork, Fiverr, Freelancer.com and data marketplaces, including my disclaimer about Data Boutique.
  • The legal split between personal, copyrighted and sui generis data versus factual data, and between non-public and public data, including the role of Terms of Service.

In the full article: The Google search trend chart, the links to the legal context post and the course tag, and the preview of the next chapter on robots.txt, sitemaps and common technologies are only in the full article.

Read the full article on The Web Scraping Club