Summary of an article published on The Web Scraping Club on 2023-02-07 (free to read). Read the full article

This is the entry point for anyone who wants to taste the water before moving to more technical material. It starts with the basics: what web scraping means, whether it is legal, and what it is typically used for. Those questions come first because most newcomers hit them before writing a single line of code.

From there it follows the order in which a project usually runs into trouble. First, what you need to get started: the Python tools and frameworks (Scrapy, Splash, Selenium, Playwright, Undetected Chromedriver) and proxies. Then the question every scraper asks sooner or later, why you get blocked. That means the common bot detection techniques, from TLS and device fingerprinting to IP blocking, and the anti-bot vendors you will meet: Akamai, Cloudflare, Datadome, F5, Kasada and PerimeterX. Each topic links to its own dedicated post.

What’s inside

  • Legal questions are split into two cases: scraping social networks like LinkedIn or Facebook, and scraping any public data you find.
  • Five Python tools get their own “what is” post, so you can compare them before choosing a stack.
  • Fingerprinting is broken down by layer: TLS, device, browser and Canvas, plus a separate post on why an IP gets blocked.
  • Each anti-bot vendor is planned with two posts, one on what it is and one on how to bypass it; only the Kasada pair is linked so far.

In the full article: The links to each individual guide on legality, use cases, Python tools and Kasada, plus the full outline of the planned anti-bot and proxy sections.

Read the full article on The Web Scraping Club