Summary of an article published on The Web Scraping Club on 2024-05-05 (free to read). Read the full article
Bright Data published The State of Public Web Data, a 90+ page report, and I took the time to read all of it. The conclusions look obvious, but the numbers say a lot about how companies see web data today. The survey was run by Vanson Bourne on 500 decision-makers at companies with 250+ employees, in the USA and UK only. That matters when you read the results. As a European, some figures don’t match my experience with local companies, though they make sense for US ones.
I go through the report by theme: web data no longer being a taboo, budgets for 2024 compared with 2023, and use across departments. I use an invented shoe company to show how web data supports strategy, e-commerce, marketing and customer service. Then I cover the link with generative AI and the push for more data, more often, depending on the use case. I also say what the report doesn’t answer: how big the web data market is. I close with why we are building Databoutique.com.
What’s inside
- Why a sample limited to the USA and UK, with companies of 250+ employees, changes how the results should be read, especially from a European point of view.
- A caveat on the average yearly budget: the report doesn’t say whether data engineering salaries are included, and that changes how big the number really is.
- How refresh frequency follows the business case, with dynamic pricing and travel fares at one end and real estate monitoring at the other.
- The only estimate of the total web scraping market I have seen, shown by Zyte at their 2023 Extract Summit, and why the report can’t fill that gap.
In the full article: The survey slides, the shoe company example in full, the Zyte market size estimate, my reading of the budget figure and the details on Databoutique.com.