Summary of an article published on The Web Scraping Club on 2022-12-11 (free to read). Read the full article

Most of us have written a scraper and then had to build the rest around it: proxies, queues, storage, monitoring, blocking countermeasures. Apify sells that whole stack as a cloud platform, and its COO, Ondra Urban, is the guest of our third monthly interview. He is a former lawyer turned developer who now calls himself the chief debugging officer.

I asked him how Apify positions itself against building on AWS, how the Apify Store marketplace is doing, and whether tougher anti-bot protection is a threat or a business model. We also talked about why the team open-sourced Crawlee and chose JavaScript first, what Crawlee does for HTTP and headless browser scraping, why most scraping spend stays in-house instead of going to bought datasets, and where proxies and machine learning are heading. The last question is about the early days of the company.

What’s inside

  • How Crawlee pairs proxy IPs, cookies and generated fingerprints into sessions, instead of rotating proxies at random.
  • Why the Apify SDK was relaunched under the Crawlee name, and what that did for adoption.
  • Ondra’s two reasons for in-house scraping dominating the market, and his view on the limits of pre-scraped datasets.
  • His expectations for proxy market consolidation and for the role of machine learning in extraction.

In the full article: Ondra’s complete answers to all ten questions, with his figures on Apify Store and Crawlee, the Python plans for the SDK and the stories from the early office days.

Read the full article on The Web Scraping Club