Summary of an article published on The Web Scraping Club on 2026-01-18 (free to read). Read the full article

Anyone who has scraped for a while has dealt with robots.txt, but the topic is now misunderstood. It started as a convention for managing crawlers. Today it sits at the center of the debate on data rights, AI training and what counts as “public” data. The file is advisory: it works as a code of conduct, not as an enforcement tool, and only cooperative bots follow it.

I go through the Robots Exclusion Protocol and the Sitemaps Protocol, then the syntax beyond Allow and Disallow. From there I cover how robots.txt serves crawl budget and SEO, and how it has ended up in the courtroom with LLM training and the AI bots some sites now block by name. Then I look at what sits around the file: robots meta tags for page-level control, and the terms of service, which is a contract and not a suggestion. I close with where the real risk for a scraping professional comes from.

What’s inside

  • Which directives go beyond Allow and Disallow: Crawl-delay, wildcard and end-of-URL matching, and how conflicting rules are resolved.
  • Why site owners care about robots.txt for crawl budget and SEO, and why it does little against malicious bots.
  • How courts have so far treated robots.txt in cases involving LLM training, and what some sites now add to block AI bots.
  • The difference between browsewrap and clickwrap agreements, and why it matters when a ToS forbids scraping but robots.txt does not.

In the full article: Annotated robots.txt examples, the HTML sample with robots meta tags and the full list of their directives, the breakdown of what a terms of service contains, and my conclusion on where the real risk lies.

Read the full article on The Web Scraping Club