Summary of an article published on The Web Scraping Club on 2026-09-13 (free to read). Read the full article
A web search tool looks like the obvious fix for an LLM’s knowledge cutoff, and many practitioners treat it as a general-purpose way to retrieve web data. I think that is a mistake. Search works in query space, through a search engine ranking you cannot inspect. Scraping works in URL space: you request a resource and get its content. If the page is blocked by an anti-bot, the model may answer from a title and a URL and you will not know.
I start with what tools are for LLMs and how the tool calling lifecycle works, from context injection to the final response. Then I explain why search tools do not fit as a RAG retrieval layer, and why raw HTML converted to Markdown is the better format for LLM input. In the second half we ran tests in chat on Claude, ChatGPT and Perplexity. We varied the prompt wording, then asked the models to open a URL behind a paywall, to see what happens under the hood.
What’s inside
- What a web search tool returns: titles, URLs, snippets, metadata, and only sometimes the full page content.
- Why a search tool cannot be the retrieval layer in a RAG pipeline, which needs an owned, pre-indexed corpus and a vector store you control.
- Why Markdown is a good serialization format for LLM context, compared with raw HTML.
- A caveat on the tests: native tool calling in chat is more probabilistic and less controllable than calling tools via API.
In the full article: The tool definition and tool call JSON examples, the chat screenshots for Claude, ChatGPT and Perplexity on both prompts, and the paywalled URL test with its results.