Summary of an article published on The Web Scraping Club on 2024-02-08 (for paid subscribers). Read the full article

Selling scraped data is a different business from freelance gigs, and the pricing model is the part that scares people. On Data Boutique, a marketplace I co-founded, datasets can go for 10 or 20 euros per download. The bet is that a low price removes the buyer who would scrape the data himself, widens the user base and encourages recurring purchases. The platform earns a fee on each sale, so its interests are aligned with sellers and buyers.

We walk through how the marketplace works, including the fixed schemas, the quality checks and the Peer Review program. Then we take a real case: Canada Goose, a listed website with no seller, protected by Kasada. We apply as a seller for the USA dataset, go through the due diligence questionnaire and read the contract. For the scraper we use the Kameleo anti-detect browser, which we already tried on Cloudflare. We set up the Kameleo client on a Windows machine on AWS and create the browser profile through its API.

What’s inside

  • How Data Boutique differs from generic marketplaces such as AWS Data Exchange and Snowflake, where you usually have to contact the vendor and negotiate a price.
  • Why datasets must follow pre-defined schemas, what that costs in flexibility, and how buyers can still ask sellers for custom work off-platform.
  • What a seller contract contains: the E0001 e-commerce schema, contract and seller IDs, the S3 delivery directory and the AWS credentials.
  • Which data is in scope: only what is 100% legal to collect, with no personal information, copyrighted material or data behind a login.

In the full article: The canadagoose.py scraper, the full Kameleo setup and profile script on GitHub, and the result of testing Kameleo against Kasada on the Canada Goose site.

Read the full article on The Web Scraping Club