Summary of an article published on The Web Scraping Club on 2026-05-10 (free to read). Read the full article
Scraping the page is only half of the job. From a news site you get a wall of text, with teasers, sponsored lines and comments wrapped around the one fact you need. A newsletter writes “just under two percent” when your column expects 1.8. Job postings give you salary ranges, “competitive compensation” or euros, and “hybrid” means something different at every company. You cannot query, aggregate or load that into a database without preprocessing.
We went through how classical NLP (spaCy, Stanford NLP, NLTK) tried to solve this and where it stops, then through LLMs as extractors and the problems they bring when used alone. The setup I propose splits the work: the LLM reads and normalizes the text, while schema libraries enforce types and reject malformed output. We compare a direct GPT-4o call with a schema-enforced one, using Pydantic and Instructor, on the same news article.
What’s inside
- Three scraping targets show what unstructured text costs you: news with signals buried in noise, financial newsletters with vague wording, and job postings with ambiguous seniority, pay and work arrangements.
- Classical NLP (NER, part-of-speech tagging, dependency parsing, relation extraction) was a real step up from regex, but it breaks on domain shift and needs extra layers to resolve context.
- LLMs used alone have three failure modes: no guaranteed output format, required fields that go missing or get invented, and hallucination. Together they make a 10,000-article pipeline hard to trust.
- The comparison inspects the type of every extracted field, so you can check what a plain prompt returns against what a defined schema enforces.
In the full article: Code for the direct GPT-4o call and for the Pydantic and Instructor version, the sample Fed article, the outputs of both runs, and the diagram of the two-step architecture.