Summary of an article published on The Web Scraping Club on 2023-10-29 (free to read). Read the full article
The Web Data Extraction Summit 2023 ended last Thursday in Dublin, and I wrote this wrap-up for those who could not be there. Web scraping looks like a niche and companies are sparse around the world, so it is hard to find an occasion to exchange ideas in person with someone in the same industry, even with competitors. That is why an in-person event still makes sense.
I go through the talks that stayed with me: AI embedded in scraping solutions, the decline of datacenter proxies, how anti-bots detect bots, the legal boundaries of scraping and of collecting AI training data, and what a marketplace specialized in web data means for buyers and sellers. I also give my own view on the size of the industry, starting from the market research shown at the 2022 edition. Speakers come from Zyte, Rayobyte and Databoutique.com, among others.
What’s inside
- My take on whether the 6 billion USD estimate for 2023 holds up after the generative AI boom
- The condition for AI in scraping tools: scalable and reliable, with the option to override its decisions
- The question from the Rayobyte talk: do you know what your scraping costs, and could your company absorb more expensive anti-bot solutions
- The comparison of a data marketplace to moving from a made-to-order artisan to mass distribution stores
In the full article: My reasoning on the market size estimate, the speaker-by-speaker comments with LinkedIn profiles, and the links to Scrapoxy and to the mobile API reverse engineering article.