Summary of an article published on The Web Scraping Club on 2024-05-26 (free to read). Read the full article

Web data is in higher demand than ever because of the AI frenzy, so this is a good time to start working as a web scraping professional. The question is what you actually need to know and in what order, and which parts of it are not code at all.

We use the lessons of the Web Scraping from 0 to Hero course as the path. It starts with the fundamentals: legal and ethical limits, how websites are built, and the checks to run before writing any scraper. Then come the tools, from proxies to unblockers and anti-detect browsers, and the technical skills: Python with BeautifulSoup, Scrapy, Selenium and Playwright, selectors, and databases for storing the data. After that we cover practice projects, ways to monetize them, building a portfolio, and the sources and communities I follow to stay current.

What’s inside

  • Legal and ethical limits come first, with a guest introduction to compliant scraping by Sanaea Daruwalla of Zyte.
  • Which tools fit which job, and where Node.js sits next to Python.
  • Where to sell what you build: freelance platforms, a dataset marketplace and a scraper marketplace.
  • A list of newsletters, forums, a wiki and Discord servers to follow once the basics are covered.

In the full article: The links to each lesson of the course, the specific resources and Discord servers I recommend, and the marketplaces named for selling datasets and scrapers.

Read the full article on The Web Scraping Club