Summary of an article published on The Web Scraping Club on 2025-10-02 (free to read). Read the full article

Autumn is the busiest season for web scraping events. Oxycon 2025 in Vilnius, where I spoke on a panel of speakers, gave a good picture of what the industry is working on. LLMs dominated, but not only: some talks showed that computer vision and NLP still solve specific tasks, and the old problems of HTML parsing, scaling and anti-bots are still there.

I wrote this wrap up while the recordings are not yet public. We saw a preview of Oxylabs’ new AI Studio, used from Cursor to get data from websites with a prompt instead of code. I describe how I use LLMs as a content creator and as a scraping professional, including writing scrapers with Cursor and MCPs. I also share my view on AI companies scraping my articles and on pay-per-crawl. Last, I cover the Oxylabs HTML parsing tool, the NielsenIQ talk on scaling, and the anti-bot panel.

What’s inside

  • Why computer vision and NLP still matter for sentiment analysis and CAPTCHA solving, according to Zia Ahmad’s talk
  • How Fred de Villamil described NielsenIQ scaling its scraping with people, processes and resource management
  • Why I see the AI-driven traffic drop for content creators as a problem I can’t measure from my own analytics
  • The case for setting realistic expectations with stakeholders when anti-bots are involved

In the full article: Our notes on each talk and panel, the AI Studio workflow shown in Cursor, my full reasoning on content creators and AI scraping, and the link to my pay-per-crawl analysis.

Read the full article on The Web Scraping Club