Summary of an article published on The Web Scraping Club on 2025-12-21 (free to read). Read the full article
If you have been scraping for a while, you know that getting data off a website is only half the job. Laws differ by country, and data is not exempt. Then there is plain etiquette: hammer a server and your IP gets banned. Being ethical and being sustainable turn out to be mostly the same thing.
I collected the five rules I consider golden. They cover robots.txt, which is voluntary but defines the house rules, and the User-Agent, where mimicking a browser is common but a descriptive string is the ethical choice. They also cover scraping at a respectful rate with delays, concurrency control and proxies, and handling errors and retries without overloading the target. The last one is the kind of data you scrape and what you use it for, with GDPR, CPRA, copyright and data sovereignty in the picture. I close with why this matters professionally and what recent court cases say about robots.txt.
What’s inside
- How to check robots.txt programmatically in Python, with Reddit’s strict file as the example.
- What a descriptive User-Agent should contain and why it gives a site admin a way to contact you before banning you.
- Why testing a new script on a few pages before scaling is part of scraping politely.
- Why recent cases involving OpenAI and Anthropic mean robots.txt has no universal legal weight, and why you should still respect it.
In the full article: The Python snippets for the robots.txt check and the delayed requests, the sample User-Agent header, the Reddit robots.txt screenshot and the links to our pieces on retries, rate limits and concurrency.