Summary of an article published on The Web Scraping Club on 2023-03-02 (for paid subscribers). Read the full article
Once a project grows to hundreds of scrapers, checking logs by hand stops working. Scrapy logs everything, but nobody reads hundreds of logs every day. Scheduling has the same problem, because each scraper has its own frequency, time window and requirements, and many of them run at once.
In this episode of The Lab I look at Scrapeops, which I found while searching for a monitoring tool for my own Scrapy fleet. It combines a monitoring and scheduling dashboard with a proxy aggregator, and the dashboard is the product we cover here. I have no commercial agreement with them. We first place it among the alternatives: Scrapy Cloud by Zyte, Scrapyd, and Cron or Task Scheduler. Then we connect scrapers to the Scrapeops dashboard, both Scrapy spiders and scrapers built on Python requests, and see how the executions show up there.
What’s inside
- How Scrapy Cloud by Zyte compares with Scrapyd and OS schedulers for running spiders on a schedule
- A way to make Cron manageable at scale by keeping the scheduling parameters in a database
- How Scrapeops connects to a Scrapyd server for scheduling, covered in an earlier post
- The two Scrapeops products, the proxy aggregator and the monitoring dashboard, and which one this episode uses
In the full article: The Scrapy settings and the setup for Python requests scrapers, plus the Scrapeops dashboard screenshots of the executions.