Summary of an article published on The Web Scraping Club on 2025-10-17 (for paid subscribers). Read the full article

HTTP is stateless, so everything a scraper needs to look like a continuing user, a login, a consent choice or a cleared anti-bot check, lives in cookies and sessions. A browser handles this for free. A script does not, and getting it wrong means failed logins, broken multi-step flows, repeated consent walls or quick detection. Anti-bots like Cloudflare, Akamai, PerimeterX and Datadome all store their verdict in cookies.

I start from the definitions and the differences between the two. Then I go through why they matter for scraping: authentication flows, consent gates such as YouTube in Europe, and the clearance cookies of the main anti-bots. We look at how a persistent client in HTTPX shares cookies across requests, compared with the Scrapy default. The last part sets up a real case on Leboncoin.fr. The idea is to get the clearance cookie once through a web unblocker and reuse it, so we don’t pay for every request.

What’s inside

  • Where cookies and sessions differ on lifetime, storage and security, and why a session usually needs a cookie to work.
  • The cookie names each anti-bot uses to store session state, and why a saved cookie may still fail when it is tied to an IP or a session.
  • How YouTube’s consent cookies, SOCS and VISITOR_INFO1_LIVE, work, and why we can’t forge them but can collect them with a browser.
  • How HTTPX behaves with plain requests versus a Client, and what the CookieJar holds after browsing several sites.

In the full article: The Leboncoin.fr case with the unblocker, the Scrapy and Camoufox scripts, the client_test.py script and the cost calculation for reusing cookies are in the full article.

Read the full article on The Web Scraping Club