Summary of an article published on The Web Scraping Club on 2024-12-19 (for paid subscribers). Read the full article
In the last episode we improved the standard Scrapy logs to print live data and summary statistics. Printing them on screen is not enough when you run hundreds of scrapers, so we want to push them to AWS Redshift. That needs ingestion with high concurrency, no bottleneck on the scrapers from database writes, and batch consumption instead of millions of small inserts. RabbitMQ covers all three.
We modify the middleware so log_metric sends messages to a RabbitMQ queue instead of the console. I go through publishers, exchanges, queues, consumers and listeners, then we install Pika, start RabbitMQ locally with Docker, and create the scrapy_logs queue from the management UI. The consumer that reads the queue and writes to the database is the next piece.
What’s inside
- How acknowledgment (ACK) and requeueing keep delivery reliable when a consumer fails to process a message.
- Why Pika fits lightweight systems like Scrapy spiders, and its support for both synchronous and asynchronous communication.
- The parameters we set on scrapy_logs: message TTL (one hour), queue expiry and maximum queue length.
- The demo runs on a local RabbitMQ install, which can move to a cluster as message volume grows.
In the full article: The modified log_metric middleware code, the queue consumer that writes batches to the database, and the 71.RABBITMQ folder of the GitHub repository.