Summary of an article published on The Web Scraping Club on 2026-09-27 (free to read). Read the full article

Platform trending tabs show only part of the picture. Conversations move across platforms, engagement changes over time, and apparent growth can be the same post reposted again and again. Catching a trend early means collecting posts and engagement metrics systematically, then separating signal from noise. Every platform exposes data differently and has its own limits and anti-bot measures, so one generic extraction strategy will not work.

We walk through the whole pipeline. It starts with source selection, official APIs versus direct scraping, and the legal side: terms of service, GDPR, CCPA and robots.txt. Then comes infrastructure: residential proxies, Playwright and Puppeteer versus HTTP clients, batch jobs versus streaming. We also cover managed scraping APIs for those who don’t want to run proxies. On the data side, we use Python to extract hashtags, mentions and keywords, track engagement over time, and clean data with deduplication and bot filtering.

What’s inside

  • How X and LinkedIn deliver data differently, through internal GraphQL endpoints and obfuscated JSON calls, and what their official APIs actually allow.
  • Why robots.txt is not legally binding on its own but can still count as evidence of bad faith, with LinkedIn’s file as the example.
  • Where regex stops working for trend signals and how NLP-based entity extraction picks up, including the vocabulary limit of that approach.
  • The trade-off in polling interval when building engagement time series: tighter intervals catch virality earlier but burn through the proxy pool faster.

In the full article: Python code for hashtag, mention and keyword extraction, engagement tracking and content-hash deduplication, plus the sample outputs and the DevTools and robots.txt screenshots for X and LinkedIn.

Read the full article on The Web Scraping Club