What it is

Web Bot Auth is a system built on RFC 9421 and HTTP Message Signatures that allows a verifier like Cloudflare to authenticate a crawler from a signing-key directory that the user publishes.

How it works

The tool checks a specified directory, which can be a domain or a full directory URL, for the existence of the /.well-known/http-message-signatures-directory. It performs a key-by-key check to identify mistakes that stop verification, including a leaked private key.

Sources


The Web Scraping Wiki is compiled from the tests and research published on The Web Scraping Club, the newsletter for web scraping professionals. Subscribe to get the next article in your inbox.