What it is
Web Bot Auth is a system built on RFC 9421 and HTTP Message Signatures that allows a verifier like Cloudflare to authenticate a crawler from a signing-key directory that the user publishes.
How it works
The tool checks a specified directory, which can be a domain or a full directory URL, for the existence of the /.well-known/http-message-signatures-directory. It performs a key-by-key check to identify mistakes that stop verification, including a leaked private key.
Related
Sources
The Web Scraping Wiki is compiled from the tests and research published on The Web Scraping Club, the newsletter for web scraping professionals. Subscribe to get the next article in your inbox.