Networking between two CTs, each with a NATed public IPv4 (hairpin NAT?)

kpv · December 23, 2021, 1:42am

I have an Ubuntu 18.04 host with LXD 3.0.3, it has 2 public IPv4. The LXD host runs 2 CTs (srv01 and srv02) connected to a bridge, each using private RFC1918 IPv4s. On the host, I manually NAT (DNAT, SNAT) one public IP to each CT (note: I fwd ports for smtp/imap/pop/web/etc separately).

Each CT runs a mail-server and I would like for them to exchange mail directly, by connecting to eachother’s tcp/25 port on their PUBLIC IPv4 as reported by DNS.

What is the recommended way to configure the LXD host’s iptables, so that it does hairpin NAT between the two CTs’ public IPv4 addresses, without possibly causing issues to LXD?

Thank you in advance, KP

$ lxc network show lxdbr0
config:
  ipv4.address: 10.114.29.1/24
  ipv4.dhcp.expiry: 192h
  ipv4.firewall: "true"
  ipv4.nat: "false"
  ipv6.address: fd42:xyz:1/64
  ipv6.nat: "true"
description: ""
name: lxdbr0
type: bridge
used_by:
- /1.0/containers/srv01
- /1.0/containers/srv02
- /1.0/containers/vm01
- /1.0/containers/vm02
managed: true
status: Created
locations:
- none

CT srv01 internal IP 10.114.29.51 and external 95.216.xx.yy
CT srv02 internal IP 10.114.29.61 and external 95.216.ww.zzz

# iptables -L -v -n --line-numbers -t nat
Chain PREROUTING (policy ACCEPT 1061 packets, 78227 bytes)
num   pkts bytes target     prot opt in     out     source               destination         
[...]
11    409K   22M DNAT       tcp  --  enp0s31f6 *       0.0.0.0/0            95.216.xx.yy         tcp dpt:80 to:10.114.29.51:80
12    124K 5743K DNAT       tcp  --  enp0s31f6 *       0.0.0.0/0            95.216.xx.yy         tcp dpt:443 to:10.114.29.51:443
13    430K   24M DNAT       tcp  --  enp0s31f6 *       0.0.0.0/0            95.216.xx.yy         tcp dpt:25 to:10.114.29.51:25
14    157K 8142K DNAT       tcp  --  enp0s31f6 *       0.0.0.0/0            95.216.xx.yy         tcp dpt:110 to:10.114.29.51:110
15    380K   23M DNAT       tcp  --  enp0s31f6 *       0.0.0.0/0            95.216.xx.yy         tcp dpt:143 to:10.114.29.51:143
16   79194 4306K DNAT       tcp  --  enp0s31f6 *       0.0.0.0/0            95.216.xx.yy         tcp dpt:587 to:10.114.29.51:587
17    138K 8263K DNAT       tcp  --  enp0s31f6 *       0.0.0.0/0            95.216.xx.yy         tcp dpt:993 to:10.114.29.51:993
18   38414 1987K DNAT       tcp  --  enp0s31f6 *       0.0.0.0/0            95.216.xx.yy         tcp dpt:995 to:10.114.29.51:995
19    1362 67902 DNAT       tcp  --  enp0s31f6 *       0.0.0.0/0            95.216.ww.zzz        tcp dpt:80 to:10.114.29.61:80
20     701 32292 DNAT       tcp  --  enp0s31f6 *       0.0.0.0/0            95.216.ww.zzz        tcp dpt:443 to:10.114.29.61:443
21     104  4884 DNAT       tcp  --  enp0s31f6 *       0.0.0.0/0            95.216.ww.zzz        tcp dpt:25 to:10.114.29.61:25
22      48  2004 DNAT       tcp  --  enp0s31f6 *       0.0.0.0/0            95.216.ww.zzz        tcp dpt:110 to:10.114.29.61:110
23      42  1752 DNAT       tcp  --  enp0s31f6 *       0.0.0.0/0            95.216.ww.zzz        tcp dpt:143 to:10.114.29.61:143
24       7   400 DNAT       tcp  --  enp0s31f6 *       0.0.0.0/0            95.216.ww.zzz        tcp dpt:465 to:10.114.29.61:465
25      63  2844 DNAT       tcp  --  enp0s31f6 *       0.0.0.0/0            95.216.ww.zzz        tcp dpt:587 to:10.114.29.61:587
26      88  4448 DNAT       tcp  --  enp0s31f6 *       0.0.0.0/0            95.216.ww.zzz        tcp dpt:993 to:10.114.29.61:993
27      59  2624 DNAT       tcp  --  enp0s31f6 *       0.0.0.0/0            95.216.ww.zzz        tcp dpt:995 to:10.114.29.61:995
28      87  5207 DNAT       udp  --  enp0s31f6 *       0.0.0.0/0            95.216.ww.zzz        udp dpt:53 to:10.114.29.61:53
29      18   732 DNAT       tcp  --  enp0s31f6 *       0.0.0.0/0            95.216.ww.zzz        tcp dpt:53 to:10.114.29.61:53
[...]

Chain INPUT (policy ACCEPT 3 packets, 193 bytes)
num   pkts bytes target     prot opt in     out     source               destination         

Chain OUTPUT (policy ACCEPT 8 packets, 614 bytes)
num   pkts bytes target     prot opt in     out     source               destination         

Chain POSTROUTING (policy ACCEPT 469 packets, 19626 bytes)
num   pkts bytes target     prot opt in     out     source               destination         
1    48513 2922K SNAT       tcp  --  *      enp0s31f6  10.114.29.51       !10.114.29.0/24       tcp dpt:25 to:95.216.xx.yy
2    28431 1706K SNAT       tcp  --  *      enp0s31f6  10.114.29.51       !10.114.29.0/24       tcp dpt:53 to:95.216.xx.yy
3     1983  119K SNAT       tcp  --  *      enp0s31f6  10.114.29.51       !10.114.29.0/24       tcp dpt:80 to:95.216.xx.yy
4    20946 1257K SNAT       tcp  --  *      enp0s31f6  10.114.29.51       !10.114.29.0/24       tcp dpt:443 to:95.216.xx.yy
5        0     0 SNAT       tcp  --  *      enp0s31f6  10.114.29.51       !10.114.29.0/24       tcp dpt:143 to:95.216.xx.yy
6        0     0 SNAT       tcp  --  *      enp0s31f6  10.114.29.51       !10.114.29.0/24       tcp dpt:110 to:95.216.xx.yy
7        0     0 SNAT       tcp  --  *      enp0s31f6  10.114.29.51       !10.114.29.0/24       tcp dpt:587 to:95.216.xx.yy
8        0     0 SNAT       tcp  --  *      enp0s31f6  10.114.29.51       !10.114.29.0/24       tcp dpt:993 to:95.216.xx.yy
9        0     0 SNAT       tcp  --  *      enp0s31f6  10.114.29.51       !10.114.29.0/24       tcp dpt:995 to:95.216.xx.yy
10     10M  887M SNAT       udp  --  *      enp0s31f6  10.114.29.51       !10.114.29.0/24       udp dpt:53 to:95.216.xx.yy
11    125K   48M SNAT       all  --  *      enp0s31f6  10.114.29.51       !10.114.29.0/24       to:95.216.xx.yy
[...]
17       0     0 SNAT       tcp  --  *      enp0s31f6  10.114.29.61       !10.114.29.0/24       tcp dpt:25 to:95.216.ww.zzz
18      47  2820 SNAT       tcp  --  *      enp0s31f6  10.114.29.61       !10.114.29.0/24       tcp dpt:80 to:95.216.ww.zzz
19     445 26700 SNAT       tcp  --  *      enp0s31f6  10.114.29.61       !10.114.29.0/24       tcp dpt:443 to:95.216.ww.zzz
20      49  2940 SNAT       tcp  --  *      enp0s31f6  10.114.29.61       !10.114.29.0/24       tcp dpt:53 to:95.216.ww.zzz
21   62960 6068K SNAT       udp  --  *      enp0s31f6  10.114.29.61       !10.114.29.0/24       udp dpt:53 to:95.216.ww.zzz

kpv · December 23, 2021, 1:49am

I also put the snippet on Pastebin:

kpv · December 23, 2021, 7:34pm

Any recommendations from the LXD experts on how do NAT hairpin for CTs on lxdbr0, when the LXD host has multiple public IPv4s ?

I have already read tens of posts at stackexchange and superuser, but they were of little help.

Do I need to mark packets as suggested in

Thank you in advance for your help,
KP

PS: My other option would be to do split DNS, which I’m more familiar with …

kpv · December 26, 2021, 1:23am

Can’t say that I am not a little disappointed that nobody came to help me, anyway I found 3 solutions (I think!) to my problem.

My first option was to use Postfix “transport_maps” to relay mails from one container to the other via its private RFC1918 IP address (and vice versa). I am familiar with Postfix, so this was easy and it worked. But it would require to manually update the maps, every time a change is made on either mailservers.
My second option was to do split DNS. Again this method worked, but added quite a bit of configuration complexity, considering I just wanted to make 2 smtp servers on the LXD bridge talk to eachother over tcp/25.
My third option was to once again try to make NAT hairpin (aka reflection) work on LXD3’s lxdbr0 bridge. As I wrote before, most of the “solutions” at stackexchange / superuse etc simply do not work (and some are blatantly wrong). Finally I found this post, which although it didn’t work for me, it explained the issues:

So finally I came up with these two NAT rules on the LXD3 host (note: two rules per CT):

iptables -t nat -A PREROUTING  -i lxdbr0 -s 10.114.29.61 -d 95.216.xx.yy -p tcp --dport 25 -j DNAT --to-destination 10.114.29.51
iptables -t nat -A POSTROUTING -o lxdbr0 -s 10.114.29.51 -d 10.114.29.61 -p tcp --dport 25 -j SNAT --to-source 95.216.xx.yy

Note: I explained the addressing in my first post:
CT srv01 internal IP 10.114.29.51 and external 95.216.xx.yy
CT srv02 internal IP 10.114.29.61 and external 95.216.ww.zzz

I’m still not quite sure if this is the final solution to the problem of LXD NAT reflection as it seems “too easy”, but sofar it seems to work for me, as I’ve sent many test mails between the 2 CT’s.

Hope this post might save someone else some time …

Merry Christmas,
KP

stgraber · December 27, 2021, 5:22pm

Hey there, sorry nobody got to answer you earlier, it’s always quite a quiet time over the holidays.
The folks working at Canonical are all off between the 17th of December and 4th of January and I suspect a lot of the usual community folks hanging out around here are similarly far away from computers