Summary of an article published on The Web Scraping Club on 2024-12-12 (for paid subscribers). Read the full article
Watching a single scraper’s runtime logs on a Grafana dashboard is nice, but it does little for an operations team running hundreds of scrapers across several machines. A local Prometheus per machine does not scale there. With that many spiders you can easily reach millions of log rows per day, and logging must not slow the scrapers down.
This is the first post of a small masterclass on logging for distributed scraping. We start from what is worth tracking and split it into live metrics and summary metrics, and tie both to monitoring, debugging and cost control. Then we choose the architecture: a message queue that decouples log production from storage. We compare it with direct database writes, API-based logging and Kafka, and explain why we picked RabbitMQ. Finally, we start building a Scrapy middleware that extracts the data, printing JSON messages to screen for now. In the next post they go to RabbitMQ.
What’s inside
- Live metrics (status codes, proxy endpoint, execution IP) support dynamic proxy rotation and early fixes, while summary metrics (requests, failures, bytes transferred) support cost estimates.
- The cost questions logging should answer: which website costs the most to scrape, and whether a site stays affordable if it needs an unblocker.
- How a message queue works: producers, consumers, acknowledgments, retries and delivery guarantees, plus the use cases where it fits best.
- Why direct database writes and API-based logging struggle at this volume, and why Kafka is more infrastructure than this use case needs.
In the full article: The Scrapy middleware code from the 70.LOGGING folder of the GitHub repository, the architecture diagram and the JSON message structure we designed for the logs.