Summary of an article published on The Web Scraping Club on 2024-12-10 (free to read). Read the full article

In 2009, scraping a website cost almost nothing. Pages were plain HTML, with no logins, no JSON and no protection. A netbook, a home connection and a small MySQL server were enough to build a global catalog of real estate listings. Writing a scraper took minutes.

Today the same job is an engineering project with its own budget line. IP blocking pushed everyone toward residential proxies, and modern sites are layers of APIs and microservices that make finding the data slow before extraction even starts.

We go through the early days, including how Google, Facemash and Airbnb started with scraping. Then we look at the defenses that changed the game and split the cost into two parts: finding the data and extracting it. We work through a proxy cost example on a large e-commerce catalog, then say who should build large-scale scraping and who should buy the data.

What’s inside

  • Honey traps get a lot of fear from clients, but in 15 years I have met one, on a small site. The worry still raises QA costs.
  • CAPTCHAs were announced as the end of scraping, then faded on e-commerce because they hurt user experience. IP blocking is what actually changed the economics.
  • AI helps with writing scrapers, but it rewards skilled engineers. The AI scraping tools we see are not built to keep costs down.
  • Storage, deduplication and enrichment are left out on purpose. I compare the scope to mining: getting the raw material out of the ground, not refining it.

In the full article: The Nike.com proxy cost calculation, the November 2024 chart of residential proxy prices per GB, the cost-reduction argument for experienced engineers and the full discussion of pre-scraped datasets on Data Boutique.

Read the full article on The Web Scraping Club