Summary of an article published on The Web Scraping Club on 2024-10-20 (free to read). Read the full article

Zyte’s Extract Summit is the one occasion of the year when the main players of the web scraping industry meet in person. This year it was in Austin on October 9th and 10th, and I could not make it. I watched all the videos on YouTube instead. The main trend is clear from the talk titles: LLMs are at the center of the stage, both as data extractors and as data users.

I summarize the LLM talks from Reworkd, Apify and Zyte, and give my view on Zyte API and why its variable cost per request appeals to me. I also cover Walmart’s talk on LLMs in data engineering. Then come the talks I would not skip outside the LLM theme: the legal panel on scraping and Charity Engine’s distributed infrastructure.

What’s inside

  • Reworkd’s answer to the input problem: a 2D rendered page that keeps the visual formatting, instead of raw HTML on large sites full of scripts.
  • What a production-ready LLM scraper needs beyond a good prompt: split the human steps of building a scraper, and test the output constantly after site changes or bans.
  • The legal panel’s distinction between browse wrap and click wrap terms of service, and what the X vs. Bright Data case says about keeping our load on target servers light.
  • Walmart’s use of LLMs to fill missing fields from unstructured text, such as color taken from product descriptions, and to run sentiment analysis on reviews.

In the full article: Embedded videos of every talk, the link to the full YouTube playlist, and my comments on each speaker’s presentation.

Read the full article on The Web Scraping Club