Summary of an article published on The Web Scraping Club on 2023-11-25 (free to read). Read the full article

Whether you are a PM, an in-house developer or a freelancer, the first question on a new scraping project is the same: how much will it cost? Knowing the real costs is what lets you charge customers enough, stay inside a budget, and understand the unit economics of your scraping activity. Difficulty of the target, refresh frequency and how long the data is needed all change the answer.

I split the costs into two parts: getting the data once and maintaining the feed over time. For each part we compare three paths: building the scraper in-house, buying a commercial solution such as web unblockers or target-specific APIs, and buying pre-scraped data from marketplaces like Datarade, AWS Data Exchange and Databoutique.com. I built cost scenarios for small, medium and large websites, then looked at which path fits one-off needs and which fits long-running projects. There is also a spreadsheet to plug in your own numbers.

What’s inside

  • Commercial solutions bill per GB of bandwidth or per request, which makes providers hard to compare, and each model suits a different kind of target.
  • Maintenance of an in-house scraper is modeled as running costs (hardware, proxies, CAPTCHA solvers) plus time spent fixing a spider that breaks.
  • With a commercial solution we no longer need resources for headful browsers, but the service cost recurs at every refresh.
  • A bought dataset can be a starting point, for example a product catalog before scraping the images, but integration work on top can change the economics.

In the full article: The cost tables for each scenario and website size, the assumptions behind the hourly rate and monthly maintenance, and the link to the spreadsheet for testing your own scraping costs.

Read the full article on The Web Scraping Club