DEV Community

Matheus de Camargo Marques
Matheus de Camargo Marques

Posted on

What if We Could Build an Erlang Cluster Over the Torrent Network?

#torrent #erlang #cluster #network

What if We Could Build an Erlang Cluster Over the BitTorrent Network?

Author: Matheus de Camargo Marques – Brazil

matheuscamarques@gmail.com · LinkedIn

🤔 Before We Start – Some Questions to Ponder

  • Have you ever tried to Node.connect/1 between two home computers behind residential NATs, only to face a wall of :nodedown and TCP timeouts?
  • Why does the Erlang distribution protocol, which works flawlessly inside data centers, become almost unusable over the public internet?
  • What if your Elixir nodes could discover each other without a central registry, traverse firewalls without manual port forwarding, and communicate securely without VPNs?
  • Is it possible to trick the BEAM runtime into believing it's using plain TCP sockets while actually tunneling everything through a P2P encrypted mesh?
  • Could we reuse the same battle‑tested technology that powers global BitTorrent (DHT, hole‑punching, BEP‑44) to connect BEAM nodes anywhere in the world?

These are the questions that led me to build JusrisOs.Peer.DistCarrier – a custom Erlang distribution carrier that replaces TCP with a secure, multiplexed, hole‑punched UDP tunnel over the Mainline DHT network.

Let's explore how it works, what challenges it solves, and how you can run a full Erlang cluster across residential connections with zero open TCP ports.

The Problem: Erlang Distribution Was Built for Data Centers

The Erlang/OTP distribution protocol (:net_kernel, epmd, :rpc) is one of the most brilliant engineering marvels of functional programming. It allows developers to spawn processes across machines, execute RPCs, and communicate between PIDs as if every node lived on the same physical machine.

However, this mechanism was born in the era of early data centers. It assumes a flat, friendly network:

  • Static IP addresses.
  • Unblocked ports (epmd on 4369 and a range of TCP distribution ports).
  • Direct, uninhibited TCP routes.

Try running Node.connect/1 across two home computers sitting behind residential NATs, ISPs using Carrier‑Grade NAT (CGNAT), or dynamic IPs, and reality hits hard:

  • epmd on port 4369 fails to resolve remote hosts.
  • Random TCP distribution ports (inet_dist_listen_min/max) get silently dropped by ISP firewalls.
  • Neither node can establish a direct TCP handshake without manual port forwarding or expensive VPN overlays.

The traditional solution is to use overlay networks like WireGuard, Tailscale, or ngrok – but these add complexity, cost, and a central point of failure.

What if we could adapt the BEAM to leverage the most battle‑tested P2P architecture in existence – the BitTorrent network?

The Core Concept: Adapting BEAM to Modern P2P

The BitTorrent ecosystem solved global residential connectivity decades ago through three primary mechanisms:

  1. Mainline DHT (Kademlia / BEP‑5): Discovers peers worldwide without a centralized tracker or static IP.
  2. STUN & UDP Hole‑Punching: Opens temporary, bidirectional UDP mappings through restrictive home routers (Cone NATs).
  3. BEP‑44: Stores and retrieves signed, mutable data (Ed25519) directly on the DHT, enabling secure key‑based peer identity.

By implementing a Custom Erlang Distribution Carrier (-proto_dist), we can trick the BEAM runtime into thinking it is talking over standard TCP sockets while silently tunneling all :net_kernel traffic through encrypted Noise sessions over hole‑punched UDP sockets.

The Architecture: Zero Open TCP Ports

Instead of opening extra TCP ports on the host machine, the carrier leverages a localized loopback bridge and multiplexed Yamux streams over UDP.

+-----------------------------------------------------------------------+
|                       Erlang / Elixir Runtime                         |
|             (Node.connect / Node.spawn / GenServer.call)             |
+-----------------------------------------------------------------------+
|
[:net_kernel / OTP]
|
+-----------------------------------------------------------------------+
|                    JusrisOs.Peer.DistCarrier                          |
|                       (-proto_dist Carrier)                           |
|                                                                       |
|   +-------------------+                     +---------------------+   |
|   | Local TCP Loopback| <--- Proxy Pump ---> |    Yamux Stream     |   |
|   +-------------------+                     +---------------------+   |
+-----------------------------------------------------------------------+
|
[Noise X25519 Handshake]
|
[Synchronous UDP Hole Punch]
|
+-----------------------------------------------------------------------+
|                   Mainline DHT / Torrent Mesh                         |
+-----------------------------------------------------------------------+
Enter fullscreen mode Exit fullscreen mode

Step‑by‑Step Flow

  1. Virtual Listener (listen/2)

    When the VM boots with our custom carrier, the listen/2 callback spawns an in‑memory GenServer (ListenServer) and an ETS table to manage incoming connection requests. No TCP port is opened on any external interface.

  2. DHT Peer Discovery

    When Node.connect(:"b@host") is executed, the carrier resolves the target peer’s public endpoint (IP:Port) via Mainline DHT and Swarm queries. If the peer is not found, it falls back to a deterministic hash of the node name (useful for testing).

  3. Synchronous UDP Hole‑Punching

    The node sends probe packets from its existing Mainline DHT UDP socket to establish a bidirectional NAT mapping on both ends simultaneously. The punch is synchronous – the carrier waits for the mapping to be effective before proceeding.

  4. Encrypted Tunneling (Noise + Yamux)

    Once the UDP path is open, an ephemeral Noise X25519 session completes a secure cryptographic handshake (forward secrecy, resistance to replay). Inside this encrypted session, a Yamux stream is allocated for the distribution traffic.

  5. Loopback Proxy Bridge