Summary of an article published on The Web Scraping Club on 2024-05-05 (free to read). Read the full article

Bright Data published The State of Public Web Data, a 90+ page report, and I took the time to read all of it. The conclusions look obvious, but the numbers say a lot about how companies see web data today. The survey was run by Vanson Bourne on 500 decision-makers at companies with 250+ employees, in the USA and UK only. That matters when you read the results. As a European, some figures don’t match my experience with local companies, though they make sense for US ones.

I go through the report by theme: web data no longer being a taboo, budgets for 2024 compared with 2023, and use across departments. I use an invented shoe company to show how web data supports strategy, e-commerce, marketing and customer service. Then I cover the link with generative AI and the push for more data, more often, depending on the use case. I also say what the report doesn’t answer: how big the web data market is. I close with why we are building Databoutique.com.

What’s inside

  • Why a sample limited to the USA and UK, with companies of 250+ employees, changes how the results should be read, especially from a European point of view.
  • A caveat on the average yearly budget: the report doesn’t say whether data engineering salaries are included, and that changes how big the number really is.
  • How refresh frequency follows the business case, with dynamic pricing and travel fares at one end and real estate monitoring at the other.
  • The only estimate of the total web scraping market I have seen, shown by Zyte at their 2023 Extract Summit, and why the report can’t fill that gap.

In the full article: The survey slides, the shoe company example in full, the Zyte market size estimate, my reading of the budget figure and the details on Databoutique.com.

Read the full article on The Web Scraping Club