Summary of an article published on The Web Scraping Club on 2024-09-29 (free to read). Read the full article

Oxylabs held the 2024 edition of Oxycon last Wednesday, its annual virtual event for new products, success stories and best practices. I attended almost the whole conference and picked three talks that intrigued me. Together they show where the industry is heading: anti-bots are more sophisticated, bypassing them costs more, and scraping many websites takes more tools, which means heavier infrastructure and messier code.

I go through the three talks. The first is the Oxylabs CTO on running scraping at scale, with a benchmark for your own stack. The second is a talk on how anti-bots use mouse movement to tell humans from bots, which ends with a new open source package from Oxylabs, Oxymouse. The third is the centerpiece, OxyCopilot, an AI-powered assistant. I compare it with the AI scraping tools I mapped a few weeks ago and say what I expect from it. I also link the earlier Lab episode on mouse movements in Playwright.

What’s inside

  • Why proxy companies push unblocker solutions: write basic scrapers with Scrapy or other browserless frameworks and delegate anti-bot bypass to a third party.
  • A short history of mouse tracking as a bot detection technique, going back to a 2018 trial involving Meta.
  • How to check in the browser developer tools whether a website tracks your mouse events.
  • OxyCopilot has two features, a custom parser builder and a request builder for Scraper API. I explain why most AI tools cover only half of the job.

In the full article: Our full take on each of the three talks, the links to the Oxymouse repository and to the earlier Lab episode on ghost-cursor and Datadome, and the comparison with the AI scraping tools map.

Read the full article on The Web Scraping Club