Summary of an article published on The Web Scraping Club on 2023-03-05 (free to read). Read the full article
Proxy providers all claim quality IPs, but the term covers very different things, and the providers themselves are moving from selling IPs to selling data acquisition. Mobile and residential IP sourcing is still the murkiest part of the industry, and AI keeps being pitched as the end of scraping as we know it. I wanted a view on these points from someone who builds the products.
For our monthly interview I talked with Aleksandras Šulženko, Scraper APIs Product Owner at Oxylabs. We covered what IP quality means beyond IP and ASN diversity, how Web Unblocker selects proxies and decides whether a response is good, and whether AI will replace web scrapers. I put my hybrid-future view to him. We also discussed how Oxylabs sources residential and mobile IPs without SDKs, how big its content team is, and a pro bono project that used scraping to detect illegal content.
What’s inside
- How he splits IP quality into infrastructure reliability and pool width and depth, and why more IPs is not always better.
- The three components behind Web Unblocker: proxy selection by ML models, dynamic fingerprinting and ML evaluation of responses with retries.
- Which parts of scraping he expects AI to take over, such as parsing and task discovery, and why he still expects an arms race between scrapers and data sources.
- Project 4beta and the Lithuanian regulator tool that scraped the national IP space, built in ten weeks.
In the full article: The complete answers to all nine questions, including the Oxylabs residential proxy acquisition practices, the first results of the Lithuanian project and the origin story of its Scraper APIs.