Summary of an article published on The Web Scraping Club on 2024-09-29 (free to read). Read the full article
Oxylabs held the 2024 edition of Oxycon last Wednesday, its annual virtual event for new products, success stories and best practices. I attended almost the whole conference and picked three talks that intrigued me. Together they show where the industry is heading: anti-bots are more sophisticated, bypassing them costs more, and scraping many websites takes more tools, which means heavier infrastructure and messier code.
I go through the three talks. The first is the Oxylabs CTO on running scraping at scale, with a benchmark for your own stack. The second is a talk on how anti-bots use mouse movement to tell humans from bots, which ends with a new open source package from Oxylabs, Oxymouse. The third is the centerpiece, OxyCopilot, an AI-powered assistant. I compare it with the AI scraping tools I mapped a few weeks ago and say what I expect from it. I also link the earlier Lab episode on mouse movements in Playwright.
What’s inside
- Why proxy companies push unblocker solutions: write basic scrapers with Scrapy or other browserless frameworks and delegate anti-bot bypass to a third party.
- A short history of mouse tracking as a bot detection technique, going back to a 2018 trial involving Meta.
- How to check in the browser developer tools whether a website tracks your mouse events.
- OxyCopilot has two features, a custom parser builder and a request builder for Scraper API. I explain why most AI tools cover only half of the job.
In the full article: Our full take on each of the three talks, the links to the Oxymouse repository and to the earlier Lab episode on ghost-cursor and Datadome, and the comparison with the AI scraping tools map.