Summary of an article published on The Web Scraping Club on 2024-04-21 (free to read). Read the full article
Web scraping is still treated as a dirty secret. The “first rule” joke even sits in the r/webscraping description. Nobody says the same about ETL, which is a tool to extract, load and transform data, and scraping is a tool for extracting data from websites. It can be used for good or for harm. It isn’t a grey area: the way you implement it decides whether it’s legal.
At Databoutique.com, our marketplace for web data, we started a collaborative map to bring the sector into the open. It catalogs the companies that extract, enrich and use web data, so people can find answers to questions like where to get residential proxies or who can scrape a YouTube channel. We split it into actors and solutions. Actors are data providers, scraping enablers, web data users and system integrators. Solutions are the specific services tied to web data.
What’s inside
- Scraping enablers include proxy providers, anti-detect browsers and API providers, and listed companies can post news and discounts that we broadcast on our channels.
- A solution names a specific service, such as Hotel Dynamic Pricing or Residential Proxy, and never a provider or a product.
- A company with a broader portfolio can be listed only if one of its services explicitly refers to web data, and one actor can be linked to several solutions.
- Submissions go through a guided form with manual approval, and staff of a listed company can claim its profile with a company email.
In the full article: Links to the actors and solutions lists, screenshots of the submission and claim flows, and the listing criteria for each type of actor.