UDP NAT Traversal

LXD
1

I have an LXD cluster with hosts distributed across multiple datacenters and a couple of different VPS providers, with each host currently running the snap/4.14 version of LXD on Debian 10 (kernel 4.19).

My aim is to use something like ZeroTier or Slack-Nebula to provide a managed overlay network between the various containers, so that there is no direct container-to-container communication, at the inter or intra-host level - other than via ZeroTier/Nebula, each container will ideally only have NATted (MASQUERADE) internet connectivity.

I currently have this semi-working with ZeroTier, however this relies on UDP hole punching (with keepalives) to create ‘direct’ peer-to-peer connections between nodes as required, which does not seem to be working. This means that all of the traffic between containers over the ZeroTier network needs to be relayed via external moons, which is certainly not ideal. While I can of course run my own moon instances to reduce the horrendous latency that this causes, I’m hoping that there is some way at the host, network or container config level to support UDP hole punching.

Any ideas?

1 Like
2

I would need to know more about the networking requirements for ZeroTier, is UDP hole punching expected to work? Do both sides see the external IP of the source of the packets?

Have you run tcpdump to see where the packets are being dropped?

3

Here’s a bit of background on Zerotier… I am just starting my journey but hope it’s helpful to spark a conversation in terms of how it would work with LXD

https://youtu.be/Bl_Vau8wtgc

4’25" mark shows the UDP hole punch and explains that it’s not necessary but can improve perf