What it is
The ESP32 functions as the hardware endpoint in an MQTT TCP proxy setup. It connects to Wi-Fi and is responsible for receiving commands and forwarding the tunneled TCP traffic to the target server.
How it works
The ESP32 subscribes to an MQTT topic to receive client-to-server bytes. A local Python script acts as an HTTP proxy on the laptop, parsing the CONNECT request, publishing an open message over MQTT, and streaming the raw TLS bytes. The ESP32 receives these messages, establishes a real TCP socket to the target host, and forwards the data.
Responses from the server are sent back through the MQTT res topic, received by the Python proxy, and relayed back to the client.
TWSC experience
Not yet tested by TWSC.
Known limitations
The ESP32 has limited resources, which restricts its ability to handle a large number of simultaneous connections. It is not suitable for loading pages in multiple tabs concurrently.
Related
Sources
The Web Scraping Wiki is compiled from the tests and research published on The Web Scraping Club, the newsletter for web scraping professionals. Subscribe to get the next article in your inbox.