Summary of an article published on The Web Scraping Club on 2026-01-04 (free to read). Read the full article
Most of us start with scripts that send one request and wait for the answer. It works for small jobs. Once “n” gets large it becomes a bottleneck, because a scraper is I/O-bound and spends nearly all its time waiting on the network. A failure halfway through a run, say at request 29 of 80, can also mean starting over.
I go through why asynchronous scraping is the right choice at scale: shorter total execution time, better use of CPU you are paying for on cloud infrastructure, and fresher data for use cases where it decays fast. Then I cover the Python toolbox: asyncio as the engine, aiohttp as the async replacement for requests, and Scrapy as the all-in-one framework. To make the difference concrete, we built the same scraper twice, with requests and with aiohttp plus BeautifulSoup, and timed both on the first ten pages of quotes.toscrape.com.
What’s inside
- What an event loop, coroutines and tasks are in asyncio, and what asyncio.gather() does.
- Why aiohttp’s ClientSession and its connection pool matter for performance, and what it means that aiohttp is a library and not a framework.
- How Scrapy handles concurrency through Twisted, its partial asyncio integration since version 2.0, and why you rarely manage an event loop yourself.
- The sync and async versions share the same parsing function, so the only thing that changes is how pages are fetched.
In the full article: The full sync and async scripts, the project setup, the timing results of both runs and the comparison between them.