Last updated: 2026-11-14

Visual maps and views

  • canvases/: JSON Canvas (.canvas) visual maps of landscapes and ecosystems
    • agentic-browsers-landscape-2026.canvas
  • views/: Obsidian Bases (.base) live queries over wiki content
    • all-entities.base — see file for details
    • anti-bot-vendors.base — see file for details
    • recently-touched.base — see file for details
    • tools-and-browsers.base — see file for details

Entities

Anti-bot systems

  • aegis — Aegis is an open-source, high-performance Web Application Firewall (WAF) and reverse proxy built in Go.
  • ai-agent-reader-page — This page is designed to present different content versions of a web page, distinguishing between what a human reader sees and what a software agent reads.
  • ai-generated-content-detection — This research addresses the challenge posed by the convergence of large language models (LLMs) and social bots, which enables malicious actors to generate human-like content at scale to manipulate the information ecosystem.
  • Akamai Bot Manager — TLS/JA3-heavy detection with behavioral JS sensor. Silent drop on mismatch. Nike.com catalog uses Akamai only; TLS impersonation sufficient.
  • akamai-v3-sensor-data — Akamai v3 Sensor Data is a component of Akamai’s bot protection system.
  • AWS WAF — Amazon WAF with JavaScript challenge. Cookie factory pattern works: browser gets aws-waf-token, Scrapy reuses it.
  • bot-detection-system — A bot detection system separates real users from unwanted automation by employing probabilistic scoring rather than relying on a single check.
  • botdeflector — BotDeflector is a captcha system developed by Martin Pronk, co-founder of Queue-IT, used to protect websites.
  • botid — BotID is an invisible CAPTCHA system developed by Vercel designed to protect against sophisticated bots without presenting visible challenges or requiring manual user intervention.
  • browser-fingerprint-scanner — The Browser Fingerprint Scanner is a tool designed to run live tests to inspect browser, device, WebRTC leak exposure, network signals, headers, GPU, DRM support, and automation indicators.
  • circuit-breaker-sage — Circuit Breaker Sage is a drop-in reverse proxy designed to prevent autonomous agents from draining API budgets by catching infinite tool loops and enforcing hard spend caps within volatile RAM.
  • claw-patrol — Claw Patrol is a security firewall designed to sit between agents and production environments.
  • client-side-bot-detection — Client-side bot detection involves running detection logic within the browser environment, which presents an asymmetry where the code executes in an environment controlled by the attacker, including their CPU, debugger, and clock.
  • Cloudflare — Multi-layer defense (TLS, JSD, Turnstile, ML scoring). Most tested anti-bot in TWSC corpus.
  • cloudflare-bot-management — BotScope is a platform designed to audit anti-agentic defenses for any website, utilizing advanced heuristics to detect anti-bot and anti-agent measures with precision.
  • crowdsec — CrowdSec is a Web Application Firewall (WAF) solution that has incorporated bot detection capabilities.
  • customhouse — Customhouse is a deterministic MCP proxy designed to block money-moving and data-egress tool calls within AI agent sessions that have received untrusted content.
  • Datadome — Behavioral analysis focus. Hermes.com is the canonical hard target. Chrome leaks automation signals that Firefox/Brave do not.
  • F5 Bot Defense — Enterprise anti-bot from F5 Networks. AI/behavioral focus. Second largest by market share (2022).
  • farmfence — farmfence is a TypeScript package that provides functions for detecting and flagging traffic originating from known bot farms, scrapers, and headless probes.
  • Google SearchGuard — The challenge Google serves on a cookieless search request. A JavaScript VM with an encrypted program collects browser signals and sets the client-side SG_SS cookie (30 s TTL). Reads 72 distinct names, 32 of them automation markers or Chrome-presence probes. Builds rotate every ~2 weeks (Aug-Sep 2026).
  • grith — Grith is a security proxy designed to supervise AI coding agents by enforcing security decisions at the operating system level.
  • hcaptcha — hCaptcha is a CAPTCHA service designed to distinguish between human users and automated bots.
  • impersonate-proxy — impersonate-proxy is a local MITM proxy implemented in Go that allows users to control various network fingerprints and headers via a single YAML configuration file.
  • jscrambler — Jscrambler is a service that provides JavaScript client-side security by obfuscating code and preventing real-time debugging and tampering.
  • Kasada — Australian company. 429 signature. Hardware fingerprinting. Zero-trust first-request philosophy.
  • llm-shield-proxy — LLM-Shield-Proxy is a stateless, zero-latency reverse proxy designed for real-time Personally Identifiable Information (PII) redaction within Large Language Model (LLM) streams.
  • midnight-mimosa-malware — Midnight Mimosa is a malware campaign discovered on low-cost Android smartphones where malicious software is embedded directly into the device’s firmware during the manufacturing process.
  • PerimeterX (HUMAN Bot Defender) — Now HUMAN Security. More IP-focused than fingerprint-focused (as of 2023). Scrapy Impersonate documented as a working bypass (2024).
  • powblock — POWBlock is a simple, high-performance, zero-dependency, system/stack-agnostic Proof of Work microservice designed to act as a defense gate against AI scrapers, bots, and DDoS attacks.
  • proof-of-human — Proof of Human is a detection system designed to classify web sessions as either human or automated.
  • proxy-botnet-malware — This malware is part of a supply-chain attack targeting Android-based car head units.
  • reCAPTCHA — Google CAPTCHA system (v1/v2/v3). v3 is invisible risk scoring. Camoufox bypasses it; open-source token extractors do not.
  • residential-proxy-observatory — The Residential Proxy Observatory is a first-of-its-kind interface designed to provide researchers with a global view of residential proxy activity and anonymized internet activity.
  • sec-ch-ua-form-factors — A new HTTP client hint supported by Chrome that describes the user’s device type (e.g., Desktop, Mob
  • selenium-headless-chrome-detection — Describes four methods to detect bots using Selenium running in headless Chrome environments.
  • sentinelmcp — SentinelMCP is a stateless enterprise policy firewall and token-cost proxy designed for the Model Context Protocol (MCP).
  • shibuya-waf — Shibuya WAF is a Web Application Firewall written in Rust that functions as a reverse proxy.
  • sibuna — Sibuna is an open-source web application firewall (WAF) designed to provide web protection against unwanted bot traffic.
  • takion-api — Takion API is a service that bypasses anti-bot walls by solving challenges server-side and returning tokens and cookies.
  • tanuki — Tanuki is an anonymization proxy built for AI-assisted security testing and bug bounty hunting.
  • telltale-bot — Telltale Bot is a service designed to detect and classify website visitors as human, AI agent, or bot.
  • ticket-js — ticket.js was an anti-bot system used by Supreme during their Spring 2021 season.
  • usehuma — useHUMA is an anti-bot startup that provides bot detection services.
  • web-bot-auth — Web Bot Auth is a system built on RFC 9421 and HTTP Message Signatures that allows a verifier like Cloudflare to authenticate a crawler from a signing-key directory that the user publishes.
  • xcancel-com-antibot-captcha — This page presents a CAPTCHA challenge requiring users to “Type the coloured characters.” It includes an image for verification and provides options for retrying the challenge or obtaining a new image if the initial attempt does not match.

Browsers

  • brave-browser — Brave is an 11-year-old, privacy-focused browser company.
  • browserbase-chrome — Browserbase Chrome is utilized for running real-browser Quality Assurance (QA) against deployed previews of projects.
  • camoufox-reverse — A Camoufox fork (WhiteNightShadow) that adds an engine-level PropertyTracer recording which DOM getters a page reads. A reverse-engineering instrument, not a stealth build. TWSC used it to watch Datadome read canvas/WebGL/navigator on leboncoin.
  • chromiumfish — ChromiumFish is a stealth Chromium fork designed to present a consistent browser identity through fingerprint hardening.
  • customized-firefox — A modified version of the Firefox browser that renames the JavaScript debugger keyword to ticket_debugger.
  • ember-browser — ember is a lightweight, headless browser written in Python designed specifically for use by AI agents.
  • fortress — Fortress is an open-source stealth browser designed to bypass various anti-bot measures encountered on websites.
  • geospoof — GeoSpoof is a browser extension and application designed to match the user’s VPN IP address to the browser’s reported location, allowing it to spoof the geographical identity of the user.
  • h5i — h5i is a lightweight, open-source headless browser implemented entirely in Rust.
  • invisible-playwright — invisible_playwright is a patched replacement for Playwright designed specifically for Firefox that is engineered to pass advanced bot detection tests.
  • kittybrowser — KittyBrowser is a web browser designed to run within a terminal pane.
  • konform-browser — Konform Browser is a stealth browser based on Firefox ESR for web scraping and automation.
  • lightpanda — Lightpanda is a browser built for AI and automation, offering faster execution and lower memory usag
  • otto — Otto is a secure remote browser automation platform designed to control real browser tabs without the need to host a browser farm or manage significant infrastructure overhead.
  • pagesieve — PageSieve is a browser extension designed for declarative web scraping.
  • rayobrowse — Rayobyte’s stealth Chromium fork. Closed-source, Docker-based, CDP compatible. C++ patching. 100% benchmark score. Beta (2026).
  • real-browser — A real browser is a tool used to actively check job boards to determine if job postings are still live and active.
  • selenium — Selenium is an open-source framework used for automating web browsers.
  • tilefinch — Tilefinch is a project focused on improving page rendering, handling large scripts, and enhancing the overall browsing experience on complex websites.

Proxy networks and tools

  • 3rd-party-proxy — The 3rd Party Proxy is recommended as the browser runtime for the TraceAIO project.
  • agent-vault-proxy — agent-vault-proxy is a tool designed to provide just-in-time API keys for AI agents and other processes routed through it.
  • antigravity-proxy — The antigravity-proxy is a local Node.js proxy designed to function as a TLS-terminating server.
  • anvil — Anvil is a proxy designed to allow GitHub-compatible tools to interact seamlessly with a Forgejo instance, making it appear as if the instance were GitHub.
  • atproxy — atproxy is a program written in Rust designed to transparently proxy TCP traffic originating from an Android application.
  • blanktrace — A cross-platform Rust CLI proxy that anonymizes browser traffic by randomizing digital fingerprints
  • bright-data — Bright Data is a residential proxy network identified in research concerning proxy software development kits (SDKs) embedded in smart TV applications.
  • clashmac — A macOS native proxy tool with real-time traffic monitoring, topology visualization, and multi-hop t
  • contabo-gmbh — Contabo is a cloud provider that rents virtual private servers and dedicated servers.
  • doh-proxy — doh-proxy is an extremely simple DNS-over-HTTPS (DoH) proxy server written in Go.
  • firehol — A community-driven project that aggregates multiple IP lists, including open proxies, Tor exit nodes
  • freesocks — FreeSocks is a VPN designed for individuals whose internet access is censored, functioning as a privacy-respecting VPN in other regions.
  • greyfox-community-edition — GreyFox Community Edition is a self-hosted AI traffic proxy and local operator console designed for teams seeking to control LLM token usage, enforce per-user limits, reuse exact cached responses, and maintain visibility into AI traffic within their own infrastructure.
  • hysteria — Hysteria is a powerful, lightning fast, and censorship-resistant proxy.
  • ipidea — A residential proxy network that enrolled 9 million Android devices into a botnet.
  • IPXO — IPv4 leasing marketplace; abuse.radar.com is its reporting surface; most static ISP proxy infrastructure traces back here.
  • kampala — Kampala is a tool for intercepting and automating web traffic.
  • keyfence — Keyfence is a local proxy designed to prevent API keys and other secrets from being exposed when making requests to Large Language Model (LLM) APIs.
  • latch — Latch is a lightweight, high-performance infrastructure proxy server designed to tunnel legacy TCP traffic securely.
  • lte-modems — The LTE modems serve as the physical source for a production residential IP proxy pool.
  • meshscrape — A service that offers a distributed IP network for web scraping to avoid blocks and rate limits.
  • mnemonic-proxy — Mnemonic Proxy is a transparent HTTP proxy designed to sit between a coding agent and various local models or OpenAI-compatible servers.
  • molehill — molehill is a tool that utilizes the Mullvad VPN service to function as a rotating HTTP CONNECT proxy.
  • momoproxy — MoMoProxy is a service that provides free and paid proxy lists for various protocols, including HTTP, HTTPS, and SOCKS.
  • mozprox — mozprox is a session-scoped, memory-safe web proxy designed for private research and threat inspection.
  • nakshguard — NakshGuard is an on-premises reverse proxy designed to detect and block runaway loops in AI agent traffic before they consume excessive API tokens.
  • ncro — Nix Cache Route Optimizer, or ncro, is a small HTTP proxy written in Rust that functions as an intermediary between the nix-daemon and configured substituters.
  • novawire — NovaWire is a fast, native debugging proxy designed to intercept, read, rewrite, and replay HTTP/HTTPS traffic.
  • nyxproxy — A tool for building a self-hosted, rotating proxy gateway using IPv6 /64 subnets.
  • pangolin — Pangolin is a peer-to-peer alternative to Cloudflare Tunnels that utilizes a client-to-site VPN combined with an edge reverse proxy to provide cloaked HTTPS traffic.
  • pingora — Pingora is an in-house HTTP proxy built in Rust by Cloudflare, designed to serve over a trillion requests daily.
  • productivityproxy — ProductivityProxy is a local macOS application that runs a proxy on the user’s machine, allowing them to define custom request and response policies using a visual graph interface.
  • proxium — Proxium is a self-hosted HTTP and SOCKS5 proxy gateway that includes an administrative user interface for managing accounts, access control lists (ACLs), TLS settings, quotas, and outgoing IP pools.
  • proxmint-free-proxy-list — This repository provides a dynamically updated and re-validated list of free network proxies supporting HTTP, HTTPS, SOCKS4, and SOCKS5 protocols.
  • purroute — Purroute is an advanced proxy server designed to act as an auto-detecting proxy router or gateway.
  • quiet-sky-privacy-proxy — The Quiet Sky Privacy Proxy is a Cloudflare Worker designed to serve as a privacy core for the Quiet Sky weather proxy.
  • residential-proxy — A residential proxy is software designed to route other people’s internet traffic out through a user’s home network.
  • residential-proxy-ips — Residential proxy IPs are IP addresses provided by networks sourced from residential devices, which are utilized for web scraping activities.
  • residential-proxy-networks — Residential proxy networks route internet traffic through real consumer IP addresses.
  • residential-proxy-services — Residential proxy services allow customers to rent Internet addresses to anonymize their web traffic or spoof their true location.
  • retroweb-2-0 — RetroWEB is a forwarding proxy designed to bridge the gap between old web browsers and modern websites.
  • sandbox-proxy — sandbox-proxy is a zero-dependency, stdlib-only Go forward proxy designed to inject real credentials, such as GitHub tokens or npm tokens, into outbound requests on the wire.
  • scrapeops — ScrapeOps is a service that offers proxy solutions and aims to simplify web scraping.
  • socklight — sockLight is a development SOCKS5 proxy featuring a live Terminal User Interface (TUI) that allows users to monitor, block, and throttle every outbound connection in real time.
  • socks5-proxy — A SOCKS5 proxy is used by systems, such as Claude Code’s sandbox, to route outbound traffic and enforce an egress allowlist.
  • squid — Squid is a network proxy server that was historically used by IT administrators to block various sites deemed unproductive or out of bounds for education.
  • stratum-proxy — The Stratum proxy is a component designed to facilitate home solo Bitcoin mining operations.
  • underclass — underclass is a local, OpenAI-compatible proxy designed to pool multiple subscriptions, such as ChatGPT/Codex and GitHub Copilot, behind a single endpoint.
  • vericto-proxy — Vericto Proxy is a transparent wire-protocol proxy designed to stop destructive SQL statements before they reach a PostgreSQL or MySQL database, requiring no code changes in the application.
  • vnc2go — VNC2Go is described as a private, in-browser service that routes traffic through a random proxy.
  • webone — WebOne is an HTTP 1.x proxy server designed to make older web browsers, media players, and messengers functional within the modern Web 2.0 environment.
  • wirepane — Wirepane is an HTTPS debugging proxy designed to be used inside Claude Code, functioning as an alternative to tools like Proxyman or Charles.

Tools and libraries

  • 4chan-captcha-solver — The 4Chan CAPTCHA Solver is a project focused on training a machine learning model to solve both the standard alphanumeric and the slider-based CAPTCHAs found on 4chan.
  • ag2b — AG2B (Agent to Browser) is a client-side agentic runtime designed to allow an agent to execute its tasks directly within the user’s browser.
  • agent-web-crawlers — Agent web crawlers are named bots and agents designed to probe, grade, index, and scrape endpoints for various purposes.
  • ai-page-audit-tool — The AI Page Audit Tool is a free utility designed to assess a webpage’s suitability for being read and quoted by large language models such as ChatGPT, Claude, Perplexity, and Google AI.
  • aiohttp — AIOHTTP is an asynchronous HTTP Client/Server for asyncio and Python, used for creating bots and scr
  • Algolia — Client-side search API. EndClothing full catalog with no anti-bot despite Akamai on site.
  • AnyCrawl — MIT-licensed LLM-ready scraping API. JSON extraction mode, MCP server. No anti-bot bypass. Comparable to FireCrawl.
  • apiaxess — apiaxess is an open-source tool designed to capture, analyze, and interact with APIs from both mobile and web applications.
  • apify — Apify is highlighted as the best provider for LLMs and Markdown generation, offering turnkey web-to-LLM scrapers and pre-built actors.
  • beautiful-code-screenshots-codeshot — CodeShot is an API toolkit designed for developers and AI agents that facilitates code screenshotting, website scraping, and link preview generation.
  • botasaurus — Python scraping framework with decorator-based API. Works locally against Cloudflare/DataDome/Kasada. Fails from server (SwiftShader exposure).
  • botdeflector-solver-chrome-extension — The BotDeflector Solver Chrome Extension is a tool designed to automatically detect and solve the BotDeflector captcha.
  • browser-use — Cloud browser automation with a low-level Chromium fork for stealth. Fully headless (81% block avoidance). Cloud runs one Firecracker microVM per session on regular EC2; $0.02/browser-hour, <400ms VM cold start.
  • bytecode-traversal — Bytecode Traversal is a method for disassembling virtual machine bytecode.
  • camoufox — Custom Firefox build. Best performer on strict Cloudflare/DataDome configs. Bypasses reCAPTCHA v3 (2025).
  • canvas-fingerprint-defender — A browser extension that defends against canvas fingerprinting.
  • cfsolve — cfsolve is a Rust library and command line tool designed to solve Cloudflare Turnstile widgets and challenge pages without requiring a full headless browser.
  • chaser-oxide — A Rust-based fork of chromiumoxide for hardened, undetectable browser automation.
  • cheerio — A library used in the Node.js environment for parsing HTML and extracting data from web pages.
  • childflow — childflow is a per-command-tree network sandbox designed for Linux environments.
  • clashmax — ClashMax is a native macOS graphical client built with SwiftUI that serves as a Mihomo proxy client.
  • cloudgrip — CloudGrip is a lightweight, self-hosted AI proxy gateway and dashboard developed using Node.js, Express, and SQLite.
  • coorl — Curl-compatible CLI that mimics Chrome’s TLS (JA3/JA4) fingerprint. No browser, no JS. TLS-layer only. Narrower CLI peer to curl-cffi (by Konstantin Lebedev).
  • crawl4ai — An open-source web crawler and scraper for LLM-friendly Markdown output.
  • curl — curl is a specific bot specimen cataloged in the crawlerzoo zoo, designed for making HTTP requests.
  • curl-cffi — Python HTTP client with TLS impersonation. Critical for hybrid scraping. Also covers scrapy-impersonate and hrequests.
  • d-nw — Dánwò is the most complete open-source end-to-end testing framework designed for the agentic era, serving as an end-to-end testing framework for both web and mobile applications.
  • danubia — Danubia is a clean text extraction API designed to take a URL as input and return the page’s main content formatted as clean markdown.
  • device-and-browser-info — A platform offering tools and articles for browser fingerprinting and bot detection.
  • dolphin-anty — Anti-detect browser for multi-account workflows. Scenarios builder, Profile Synchronizer, REST API. Free tier (10 profiles).
  • dzt-s-captcha-harvester — dzt’s captcha-harvester is a tool designed for captcha harvesting, specifically to store the g-recaptcha-response token for use during checkout processes.
  • elcapture — elcapture is a native HTTPS debugging proxy designed for Mac users.
  • envoy — Envoy is an infrastructure component discussed in the context of Reddit’s feed serving path.
  • esp32 — The ESP32 functions as the hardware endpoint in an MQTT TCP proxy setup.
  • facebookexternalhit — A specific user agent string used by the Facebook crawler to retrieve information about shared conte
  • feedstock — A high-performance web crawler and scraper for TypeScript, powered by Bun and Playwright.
  • fender — fender is a transparent Docker Unix socket proxy designed to eliminate the lock-in to the implicit Docker Hub registry.
  • fingerprinterjs — FingerprinterJS v2.0 is a browser fingerprinting and bot detection tool.
  • fpscanner — A lightweight browser fingerprinting library for bot detection.
  • ghost-cursor — Bezier/Fitts’s Law mouse movement for Playwright. OxyMouse is a newer alternative with Gaussian and Perlin noise options.
  • ghx — ghx is a caching proxy designed for the GitHub CLI that utilizes singleflight coalescing to ensure that repeated commands are executed instantly.
  • go-http-client — An HTTP(s) client implemented in Golang used to make HTTP(s) requests from a Golang program.
  • gologin — Anti-detect browser, Orbita engine. Top benchmark performer 223/260 in 2024.
  • google-browser-scraper — google-browser-scraper is a Python tool designed to scrape Google search results by utilizing a real browser session routed through a user-provided sticky proxy.
  • google-search-guard-vm-reversing — MIT Node.js toolkit (Ihar Kazlouski, Aug 2026) that tears down Google SearchGuard offline: a Proxy probe that lists every browser property the challenge reads, and a VM pipeline that lifts the interpreter, profiles opcodes, traces execution and recovers the plaintext record before SG_SS encryption. Does not generate tokens.
  • goscrapy — A high-performance web scraping framework for Go, designed with the familiar architecture of Python’
  • groxy — Groxy is a small Go library designed for building forward proxy servers.
  • guestlist-tools — guestlist is a free Python library and HTTP API designed to check whether AI agents and computer-use agents can access specific websites.
  • hn-attention-cliff — hn-attention-cliff is a repository containing the scraper, analysis, and data used to measure the half-life of Show HN launches on Hacker News.
  • hodor — Hodor is a tiny reverse proxy written in Rust designed to gate web applications behind a single shared password.
  • hRequests — TLS-fingerprinted HTTP with embedded headless browser. Akamai/Cloudflare/PX pass; DataDome/Kasada fail.
  • icloud-hme — A command-line interface (CLI) tool designed to automate the generation of iCloud HideMyEmail aliases.
  • iherb-cli — A Rust command-line tool for querying product data from iHerb using a headless browser.
  • impersonate-http — impersonate-http is a Go http.Client designed to mimic the TLS handshake and HTTP/2 frames of a real browser.
  • intuned-agent — Intuned Agent is an agent designed to build, deploy, and maintain production-ready Playwright code for scrapers, crawlers, and RPA automations.
  • is-antibot — A library that detects antibot and CAPTCHA challenges from 30+ providers using signals.
  • ja3proxy — Go-based TLS Client Hello rewriter via uTLS.
  • jevcrawl — Jevcrawl is an MCP server and CLI tool designed to crawl websites specifically for AI agents.
  • Kadoa — Commercial AI scraping workflow platform. UI-driven, anti-bot included, 5 proxy locations.
  • kameleo — Anti-detect browser. Junglefox + Chroma. First to support Selenium + Playwright + Puppeteer simultaneously.
  • kimurai — A Ruby-based web scraping framework that uses AI to assist in data extraction.
  • layer3-intel — Layer3 Intel is a security firm that launched a Residential Proxy Observatory.
  • libretto — A toolkit for building robust web integrations and maintaining browser automations.
  • linkedinbot — A scraper used by LinkedIn to gather information about shared URLs in posts or messages.
  • llmcap — LLMCap is a service designed to enforce hard dollar caps on LLM API calls across various providers.
  • lucidextractor — An enterprise web scraping and SEO analysis platform.
  • lumi-geo — Lumi GEO is a free Chrome extension designed to analyze a webpage to show the difference between what AI crawlers see and what a human user sees, providing code fixes to improve visibility.
  • makraexafirecrawl — MakraExaFirecrawl is a work-in-progress implementation designed for a continual learning harness focusing on Browser-use.
  • masterhttprelayvpn — A domain-fronted HTTP/SOCKS5 proxy tool that tunnels traffic through Google Apps Script for scraping
  • math-tanh — The implementation of the Math.tanh function varies between operating systems, which serves as a subtle signal for browser fingerprinting.
  • medium-good-words-solver — Medium Good Words Solver is a Chrome extension designed to automate the process of playing and solving the “Good Words” game on Medium.
  • mitmcloak — mitmcloak is a drop-in mitmproxy addon designed to replace mitmproxy’s upstream leg by mirroring the client’s real TLS and HTTP/2 fingerprints.
  • mitmproxy — mitmproxy is a small interceptor written in Python used to build a proxy, exemplified by the simple-proxy project.
  • mitty — Mitty is a transport-agnostic proxy RPC designed for executing method chains from any isolated context, such as Web Workers, browser Tabs, or Servers, utilizing lazy evaluation.
  • mochi-js — mochi.js is a Bun-native, raw-CDP browser automation framework designed to create relationally-coherent fingerprints.
  • mycel — Mycel is a fast, decentralized web crawler, indexer, and search engine implemented in a Rust binary.
  • mysysinfo-api — The MySysInfo API is a tool that provides system information gathered from the user’s browser and device, including details such as the IP address, operating system, browser version, screen resolution, and device memory.
  • nodejs-based-scraper — A previous project mentioned in the series used NodeJS to collect temporary phone numbers and messag
  • Nodriver — Chrome automation without WebDriver layer. Passes CDP tests natively but cannot forge hardware fingerprint.
  • obscrd — An open-source content protection system for React that scrambles HTML and blocks AI crawlers.
  • obscura — A headless browser engine written in Rust for web scraping and AI agent automation.
  • open-bullet-2 — A credential stuffing tool that can be used for web scraping and automation.
  • owl-browser — A self-hosted browser automation engine for automation at scale with 256 parallel contexts and undet
  • pg-interleave — pg-interleave is a wire-protocol proxy for PostgreSQL.
  • playwright — Microsoft browser automation. Detectable by default, patchable via Patchright/Undetected Playwright.
  • propylea — Propylea is a sovereign, ultra-low-memory L7 reverse proxy, multi-domain SNI TLS multiplexer, and active perimeter defense gateway written in pure Rust.
  • proxelar — A Rust-based MITM proxy for intercepting and modifying HTTP/HTTPS traffic.
  • proxy-benchmark — proxy-benchmark is a measurement harness designed to test various combinations of proxies, browser engines, and scraping targets to diagnose request failures.
  • proxy-scraper-cli — proxy-scraper-cli is a command-line tool written in Python designed to scrape and verify free HTTP, SOCKS4, and SOCKS5 proxies.
  • proxy-server — A service that brokers connections between the browser and phone.
  • proxyboy — ProxyBoy is a Windows-native HTTP/HTTPS debugging proxy designed to capture, inspect, and modify network traffic, similar to tools like Charles Proxy or Proxyman.
  • puppeteer — A Node.js library which provides a high-level API to control Chrome or Chromium over the DevTools Pr
  • puppetflow — Puppetflow is a self-hosted browser automation platform designed for building, running, scheduling, and monitoring JavaScript workflows.
  • pw-repl — pw-repl is a Playwright-powered tool designed to allow users and AI agents to share control over a Chromium browser session through command-line interaction.
  • pydoll — Async CDP Chrome automation. Stability issues in 2026 benchmarks.
  • pyscrappy — PyScrappy is an AI-native web scraping toolkit designed to transform websites into structured, LLM-ready data.
  • python-requests — A Python HTTP library for making HTTP(s) requests.
  • qtrouter — QTRouter is a tool identified by the FBI as part of a network of proxy devices used by a Chinese state-sponsored hacking group.
  • rama — Rama is a framework designed to support various proxy protocols and configuration methods for network clients.
  • reader — An open-source, production-grade web scraping engine built for LLMs.
  • relinos — Relinos is a programmable browser that functions as a REST API, enabling users to render and interact with websites.
  • resurf — Resurf is a deterministic and reproducible test environment designed for systematically testing AI browser agents against realistic, stateful environments.
  • ricci-flow-ai-web-scraper — A Chrome extension for AI-powered web scraping that extracts data from websites and exports it to Ex
  • roxy — Roxy is a feature-rich RSS proxy written in Go that is designed to combine multiple RSS feeds into a single, queryable feed.
  • runo — Runo is an open-source tool designed to extract structured, typed JSON from any URL based on a schema defined by the user.
  • rustwright — Rustwright is a browser automation library designed for Python and Node.js that maintains the familiar Playwright API while driving Chromium using a native Rust engine.
  • ScrapeGraphAI — LLM-powered scraping library and commercial API. Non-deterministic, best for horizontal multi-site extraction.
  • scrapingduck — A scraping API that handles IP rotation, JavaScript rendering, and provides a free tier for web scra
  • scrapingsandbox — A web scraping sandbox for learning and testing web scraping techniques.
  • scrapling — Python library with three fetchers (static, dynamic, stealth). 1735x faster than BS4. Fetcher class achieves 100% on Nike/Akamai without browser overhead.
  • Scrapoxy — Open-source proxy aggregator. Unifies providers and cloud VM egress. ~20% cost savings vs. pure cloud.
  • scrapy — Scrapy is a Python library that is used in typical web scraping setups.
  • sensecollect — SenseCollect is a self-service platform designed to simplify web data extraction for lead generation and market intelligence.
  • shellroute-cli — The shellroute CLI is a command-line interface designed to route terminal commands through a configured proxy.
  • simple-selenium-chrome-crawler — A Python script that uses Selenium and Chrome to crawl web pages and extract links.
  • slick-search-engine — Slick is a customizable search engine designed to give users control over their search results.
  • socks5-bridge — socks5-bridge is a local HTTP-to-SOCKS5 proxy bridge designed for Chrome.
  • softpasskey — softpasskey is a cross-platform command-line tool designed to register a fake passkey and log users into websites without requiring a physical security key.
  • spidersuite — A powerful web crawler and security testing tool for penetration testers and security researchers.
  • spidra — A web scraping platform that uses AI to extract data from websites.
  • stalker — Stalker is a tool designed to check and report various browser and network fingerprints for privacy and analysis.
  • stop-bots — stop-bots is a TUI and a CLI tool written in Rust designed to configure web servers and firewalls to block malicious bots without requiring the user to hide behind a Content Delivery Network (CDN).
  • stunt — stunt is a local simulator that spins up stateful, realistic stand-ins for public APIs, serving as mock or stub servers for testing purposes.
  • supercrawl — SuperCrawl is an open-source web crawler implemented in the Rust programming language.
  • supreme-bot — A Supreme bot is a tool designed to automate the process of purchasing items from the Supreme website.
  • tadpole — A language for writing declarative, modular scraping code.
  • transparenttorproxy — A Linux CLI utility that transparently routes all system traffic through the Tor network using nftab
  • twitter-generator — A Python module designed to reverse-engineer and replicate X (formerly Twitter) login headers for automation.
  • uaparser-js — A JavaScript library recommended for parsing user agents to extract browser and OS information.
  • undetected-chromedriver — Patched Selenium. Stagnant repo, succeeded by Nodriver/Zendriver.
  • webassembly-simd — WebAssembly SIMD refers to Single Instruction, Multiple Data operations introduced to the WebAssembly specification.
  • webcrack — Webcrack is a tool designed to assist in the reverse-engineering of web pages, specifically by aiding in the deobfuscation of JavaScript snippets.
  • who-1ms-dog — 1ms.dog is a website that tests and reveals browser fingerprinting signals to users.
  • wick — Wick is an open-source tool that allows AI agents to access web pages as if they were real browsers.
  • wmimo — Wmimo is a modern, cross-platform GUI client developed using Flutter, designed for managing Clash and Mihomo proxy configurations on both desktop and mobile platforms.
  • wxpath — wxpath is a Python library for declarative web crawling using XPath.
  • x-twitter-ui-metrics — x-twitter-ui-metrics is a GitHub repository containing Node.js, Go, and Python implementations designed to reconstruct and bypass X (formerly Twitter)‘s UI metrics login flow challenge.
  • zenrows-batch — Zenrows Batch is an asynchronous scraping API designed to manage large URL lists as a single, managed job.

Web Unblockers

  • Oxylabs Web Unblocker — 96/100 overall. Cheapest on Kasada benchmark ($0.10/101 URLs). Best dashboard.
  • Smartproxy Site Unblocker — 80/100 overall. $12/GB. Kasada improved from 0% (2023) to 92% (2024).
  • Zyte API — 100/100 with browser rendering. Fails Kasada. Per-request dynamic pricing. Scrapy integration via scrapy-zyte-api.

Concepts

  • Agent Sandboxing — When an AI agent can execute arbitrary code (run Python, shell commands, write files), it can read anything on the machine it runs on: environment variables, API keys, database credentials, internal services. Agent sandboxing is the architecture that isolates that code execution from the infrastructure and secrets it must never reach. There are two structural patterns, documented by Browser Use in February 2026 from running millions of web agents.
  • AI Web API Fingerprinting — Chrome’s on-device AI Web APIs (Translator, Language Detector, Summarizer, and the upcoming LanguageModel) only function on hardware that meets specific minimum specifications. Querying whether these APIs are available, and measuring how fast they run, turns them into a hardware capability probe. The signal is harder to spoof than declarative properties like navigator.hardwareConcurrency because it reflects what the device can actually do, not what it claims. Documented by Anthony Manikhouth (DataDome R&D) in April 2026.
  • ai-scraping-assistants — AI scraping assistants are setups where a large language model is embedded into the development workflow to generate, modify, or repair scrapers — rather than to extract data at runtime. The LLM acts as an accelerator for the developer, not as a replacement for the scraper itself. The canonical current form is an AI-powered IDE (Cursor) connected to custom tooling via the Model Context Protocol (MCP).
  • Anti-Detect Browsers — Anti-detect browsers are Chromium or Firefox forks engineered to present a consistent, plausible fingerprint from a database of real device profiles. Unlike standard browser automation which exposes the actual hardware and software environment of the machine running the scraper, anti-detect browsers attach a pre-built fingerprint from a real consumer device to the automation session, regardless of the actual underlying hardware.
  • API Scraping — API scraping is the practice of identifying and replicating the internal HTTP requests that a browser or app makes to fetch data, bypassing HTML parsing entirely. The data is already structured, usually JSON, and the endpoint is observable by monitoring network traffic in DevTools or a proxy tool.
  • Bot Detection — Bot detection is the umbrella discipline of identifying automated traffic on a website and distinguishing it from legitimate human visitors. It sits opposite to web scraping: the same fingerprints, behaviors, and protocol-level traces that scrapers try to hide are what detection systems look for. Reading the detection literature is the most direct way for a scraper engineer to understand which signals actually leak.
  • Browser Privacy Fingerprinting Defenses — Apple, Mozilla, and Brave have shipped browser-level anti-fingerprinting protections that partition storage and add noise to or remove the very APIs that bot detection vendors relied on for high-entropy device identification. The consequence for both tracking and bot detection is that classic client-side fingerprinting is becoming low-entropy: previously distinctive signals now return generic or randomized values across users. Documented by Anthony Manikhouth (DataDome R&D) in December 2025.
  • browser-fingerprinting — A stateless, cookieless tracking mechanism that identifies browsers by collecting and hashing the outputs of browser APIs. Unlike cookies, fingerprints require no storage on the client and persist across sessions, incognito modes, and cookie clears.
  • canvas-fingerprinting — Canvas fingerprinting is a stateless tracking technique that identifies a device by drawing graphics off-screen and hashing the resulting pixels. Two machines asked to render the same text and shapes produce subtly different pixel output, because the final image depends on the GPU, the graphics driver, the font rasterizer, and the anti-aliasing settings of that specific machine. The hash of those pixels becomes a high-entropy identifier that needs no cookie and survives incognito mode and cache clears.
  • cdp-detection — A detection technique that identifies browser automation by observing the Chrome DevTools Protocol (CDP) commands sent to the browser during page load. CDP is the underlying control protocol used by Playwright, Puppeteer, Selenium (on Chromium), and related frameworks. Anti-bot systems can detect its presence independently of any JavaScript-level signals.
  • Certificate Pinning — Certificate pinning is a security mechanism in mobile apps that causes the app to reject any TLS certificate that does not match a specific expected certificate or public key, regardless of whether the device’s OS trusts the certificate authority. It prevents man-in-the-middle inspection even on a device where the inspector’s root CA has been installed.
  • cookie-session-reuse — A technique where anti-bot clearance cookies obtained through one browser session are extracted and reused across multiple subsequent requests made by a browserless HTTP client. The browser absorbs the cost of the challenge once; the HTTP client amortizes that cost across many requests.
  • homepage-first-navigation — A navigation pattern where a scraper visits the site’s homepage (or another entry-point URL) before accessing any deep content pages. The purpose is to establish session state — cookies, fingerprint checks, and anti-bot challenges — in the same sequence a real user would encounter them.
  • HTTP Performance in Scraping — HTTP performance optimization for scraping covers the set of techniques that reduce execution time, bandwidth cost, and infrastructure requirements without changing the data extracted. The main levers are concurrency, protocol selection, caching, and retry behavior.
  • hybrid-scraping — An architecture that uses a full browser only for the portions of a session that require anti-bot clearance, then hands off to a lightweight HTTP client for all subsequent data requests. The browser handles authentication and challenge resolution; the HTTP client handles scale.
  • inventory-tracking — Inventory tracking through web scraping is the practice of monitoring the stock levels of products on e-commerce sites and retail websites by extracting data that those sites expose for operational purposes — usually to power their own availability and pickup features. The extracted data is most valuable as a repeated time series rather than a single snapshot.
  • JavaScript Proxy tracing — A reverse-engineering technique that enumerates every browser property an anti-bot script reads without reversing the script. The script is executed with a JavaScript Proxy standing in for the global object, so each property read, in check and function call is logged, including reads of names that do not exist, which is exactly where automation probes live and what an ordinary getter override cannot catch.
  • llm-scraping — Using a large language model to extract structured data from web pages, replacing or supplementing traditional CSS selector or XPath-based parsing. The LLM interprets the page content — either raw HTML, cleaned text, or a screenshot — and returns structured output matching a defined schema. A separate but related use case is using LLMs to generate scraper code rather than to parse data at runtime.
  • location-data-scraping — Location data scraping covers the techniques for extracting data that is spatially indexed — store locators, accommodation listings, restaurant maps, delivery zones — where the API requires geographic inputs (coordinates, radius, or bounding box) and returns data for the area specified. The central challenge is ensuring complete geographic coverage without issuing redundant or missing queries.
  • ML-Based Bot Detection — ML-based bot detection refers to the use of machine learning models to classify web traffic as human or automated. Unlike rule-based systems (IP blocklists, simple rate limits, user-agent checks), ML models learn from behavioral patterns across large traffic datasets and can identify subtle automation signals that static rules miss.
  • Mobile App Scraping — Mobile app scraping is the practice of intercepting the network traffic generated by a mobile app and replicating the underlying API calls in a scraper. Since mobile apps communicate with backends through the same HTTP/HTTPS protocol as browsers, the data they fetch can be captured and reproduced.
  • Mobile Proxy Farming — Mobile proxy farming is the practice of building and operating infrastructure that routes proxy traffic through real cellular network connections — typically USB modems or SIM-equipped hardware — to produce genuine mobile IP addresses with CGNAT-level trust properties. Unlike SDK-sourced residential proxies, which borrow device connections from consumer apps, a proxy farm gives the operator full control over the hardware, IP rotation, and connectivity.
  • Mouse Movement Emulation — Mouse movement emulation is the practice of generating pointer trajectories that match the statistical properties of human hand movement during browser automation. Anti-bot systems that monitor mouse events can distinguish between the instantaneous or straight-line movements produced by automation frameworks and the curved, velocity-varied paths that humans produce. Emulation attempts to make automated mouse behavior indistinguishable from real user input at the event listener level.
  • Prerender Stealth Request — A <link rel="prerender"> element in Chromium-based browsers fires a real HTTP request that bypasses Content Security Policy, does not appear in the DevTools Network tab, and sends the browser’s real User-Agent even when DevTools’ Network Conditions UA override is active. Discovered by Manuel Garcia (brokenbrowser.com) in May 2026, independently confirmed by soutag (who found it earlier but did not publish).
  • proxy-fundamentals — Proxies route outbound HTTP requests through an intermediate IP address, changing the apparent origin of the request. In a scraping context, the proxy’s IP reputation, ASN, and network type are often more consequential than any browser-level signal.
  • Scraping Infrastructure — Scraping infrastructure encompasses the compute, storage, scheduling, monitoring, and proxy management systems that run scrapers reliably at scale. The choice of infrastructure directly determines cost structure, IP rotation capability, and operational overhead. A scraper that works on a laptop fails at 100,000 requests per day without the right infrastructure beneath it.
  • scraping-economics — The economics of web scraping encompass the cost structure of running scraping operations, the business models for monetizing scraped data, and the market dynamics of the web data industry. The industry spans two distinct roles: those who sell the infrastructure and tools that enable scraping (“selling shovels”), and those who scrape and sell the resulting data.
  • Static ISP Proxies — Static ISP proxies are a hybrid category between datacenter and residential proxies. The IP addresses belong to real ISP allocations (Comcast, RCN, AT&T, Verizon) or to fabricated ISPs with fictional names. In both cases the server handling traffic is physically in a datacenter. The distinguishing characteristic is that the address block carries ISP reputation rather than datacenter ASN reputation, and the IP stays assigned to a single customer for weeks or months rather than rotating.
  • tls-fingerprinting — A detection technique that identifies the client making an HTTPS request by analyzing the structure of its TLS ClientHello message, before any application-layer data is exchanged. The most common implementation is JA3, which produces an MD5 hash from five fields in the handshake: TLS version, cipher suite IDs, extension IDs, elliptic curve IDs, and elliptic curve format IDs. JA4 is the more recent successor scheme, more granular and harder to evade, but built on the same principle of hashing handshake fields the client never chooses from code.
  • VM-based JavaScript obfuscation — An anti-bot delivery pattern where the detection logic is not present in the JavaScript shipped to the browser. The page contains an interpreter (a virtual machine written in JavaScript) and an encrypted bytecode program. The checks, the fingerprint collection and the token assembly all happen inside the bytecode. Searching the script text for navigator.webdriver finds nothing because the string is built at runtime by the VM.
  • Web Unblockers — A web unblocker is a managed API that accepts a target URL and returns the page content after transparently handling IP rotation, CAPTCHA solving, browser fingerprinting, and anti-bot bypass on the provider’s infrastructure. The caller makes a standard HTTP request; the provider does the work of getting through the target’s defenses.
  • web-scraping-legal-landscape — The legal landscape for web scraping is a contested, evolving body of case law and regulation across multiple jurisdictions. No single statute governs it. Whether any particular scraping operation is legal depends on the interaction of: the Computer Fraud and Abuse Act (CFAA), the Digital Millennium Copyright Act (DMCA), copyright law, contract law (Terms of Service), and in Europe, the GDPR. The area has shifted materially between 2022 and 2026, with courts increasingly being asked to define where public data access ends and illegal circumvention begins.
  • WebDriver vs CDP vs WebDriver BiDi — Three distinct protocols exist for controlling a browser programmatically. They differ in design origin, transport mechanism, browser support, and what detection surface they expose. Choosing the wrong one for a scraping context can introduce detection vectors that are independent of any fingerprint or behavior emulation.
  • webrtc-ip-leak — A WebRTC IP leak is when a page learns a client’s real LAN or WAN IP address through the WebRTC API, even though the client’s HTTP traffic is routed through a proxy. It happens because WebRTC gathers network candidates over UDP, on a path that an ordinary HTTP proxy does not control. For a scraper this is a coherence failure: the site sees one IP for the page requests (the proxy) and a different IP for WebRTC (the real machine), and the mismatch is a strong automation and evasion signal.
  • WebSocket Bot Detection — WebSocket bot detection refers to anti-bot techniques that exploit the stateful, persistent nature of the WebSocket protocol to identify automated clients. Because WebSocket connections are long-lived and bidirectional, servers can monitor client behavior continuously over the connection lifetime, enabling detection patterns that are not possible with stateless HTTP.
  • websocket-scraping — WebSocket scraping is the practice of connecting to a site’s WebSocket endpoint directly, subscribing to the same channels the browser subscribes to, and receiving the same real-time data stream. Unlike HTTP scraping, which is request-response, WebSocket connections are persistent and bidirectional. The server pushes updates without the client polling.

Comparisons

  • Anti-Detect Browser Benchmark 2024 — Eight commercial anti-detect browsers evaluated against a standardized fingerprint detection test suite. The comparison measures how convincingly each browser impersonates a real consumer device when probed by the same fingerprinting methods that anti-bot systems use.
  • Camoufox vs its forks — After development of Camoufox moved to CloverLabs in 2026, its public repository grew past 750 forks. Most are mirror bots with no commits of their own. This page compares stock Camoufox against the three forks that actually change anti-detect behavior, and records which one to run against a real target. The comparison is grounded in our 2026-06 testing against Datadome on leboncoin.fr, not on the forks’ stated goals.
  • Firefox-based vs Chrome-based Stealth Tools — Firefox-based stealth tools (primarily Camoufox) against Chrome-based stealth tools (Pydoll, undetected-chromedriver, Patchright) for bypassing anti-bot systems. This comparison matters because the browser engine choice has a measurable impact on bypass success rates, independently of the stealth techniques applied on top.

Timelines

  • Cloudflare Bypass Evolution — How Cloudflare bypass techniques have evolved as Cloudflare has progressively tightened its defenses. This timeline tracks what worked, what stopped working, and what replaced it, based on TWSC’s direct testing.