Summary of an article published on The Web Scraping Club on 2023-03-26 (free to read). Read the full article
Every month I collect what mattered in the web scraping industry, so you don’t have to follow every vendor blog. March had a few items worth a practitioner’s attention. The big names in the proxy and data collection business joined forces on shared ethical rules. Two tools also shipped that change how some of us build scrapers and handle anti-bot challenges.
I cover the Ethical Web Data Collection Initiative, whose founding members include Zyte, Smartproxy, Rayobyte and Oxylabs, and the areas its principles cover. In the tech updates I look at Apify’s new Python SDK, which I plan to test, and at Bright Data’s Scraping Browser, a managed browser session you call from Puppeteer or Playwright projects. I also say where the make-or-buy question comes in. The recap closes with a video on monetizing scrapers on the Apify marketplace and the most-read article of the month.
What’s inside
- The initiative sets rules in four areas: legality, ethics, ecosystem management and social responsibility.
- Apify’s SDK works with Requests, HTTPX, Beautiful Soup, Playwright, Selenium and Scrapy, and Actors can be published and sold on the Apify marketplace.
- The Scraping Browser is meant to remove the pain of finding which browser and configuration work on a given site, at a price that makes it a make-or-buy decision.
- A Scraping Browser test against the best-known anti-bot solutions is planned, following the earlier Web Unlocker hands-on.
In the full article: The links to the EWDCI Principles 1.0 document, the Apify and Bright Data documentation, the video on publishing Actors and the most-read March hands-on article.