Summary of an article published on The Web Scraping Club on 2022-12-11 (free to read). Read the full article
Most of us have written a scraper and then had to build the rest around it: proxies, queues, storage, monitoring, blocking countermeasures. Apify sells that whole stack as a cloud platform, and its COO, Ondra Urban, is the guest of our third monthly interview. He is a former lawyer turned developer who now calls himself the chief debugging officer.
I asked him how Apify positions itself against building on AWS, how the Apify Store marketplace is doing, and whether tougher anti-bot protection is a threat or a business model. We also talked about why the team open-sourced Crawlee and chose JavaScript first, what Crawlee does for HTTP and headless browser scraping, why most scraping spend stays in-house instead of going to bought datasets, and where proxies and machine learning are heading. The last question is about the early days of the company.
What’s inside
- How Crawlee pairs proxy IPs, cookies and generated fingerprints into sessions, instead of rotating proxies at random.
- Why the Apify SDK was relaunched under the Crawlee name, and what that did for adoption.
- Ondra’s two reasons for in-house scraping dominating the market, and his view on the limits of pre-scraped datasets.
- His expectations for proxy market consolidation and for the role of machine learning in extraction.
In the full article: Ondra’s complete answers to all ten questions, with his figures on Apify Store and Crawlee, the Python plans for the SDK and the stories from the early office days.