Device add NAT issue, fails to allow connection, but worse it breaks the IP stack hours later

LXD
1

Hi,

I have been trying to make a proxy device which is optimised proxying via NAT from the manual:
https://lxd.readthedocs.io/en/latest/containers/#type-proxy
snap 3.17
ubuntu 18.04 host

convert the DHCP assigned IP address to a static address

lxc config device override c1copy eth0 ipv4.address=10.250.231.11

add a device for NAT (ip address chnaged)

lxc config device add c1copy c1copyProxy proxy listen=tcp:IPaddress:7777 connect=tcp:10.250.231.11:5432 nat=true

device added, good.

lxc config device show c1copy:
c1copyProxy:
connect: tcp:10.250.231.11:5432
listen: tcp:IPaddress:7777
nat: “true”
type: proxy

and iptables -t nat -L shows lines added.

Chain PREROUTING (policy ACCEPT)
target prot opt source destination
DNAT tcp – anywhere (FQDN) tcp dpt:7777 /* generated for LXD container c1copy (c1copyProxy) */ to:10.250.231.11:5432

Chain INPUT (policy ACCEPT)
target prot opt source destination

Chain OUTPUT (policy ACCEPT)
target prot opt source destination
DNAT tcp – anywhere (hostname) tcp dpt:7777 /* generated for LXD container c1copy (c1copyProxy) */ to:10.250.231.11:5432

Chain POSTROUTING (policy ACCEPT)
target prot opt source destination
MASQUERADE all – 10.250.231.0/24 !10.250.231.0/24 /* generated for LXD network lxdbr0 */

but ss-ltr shows no port 7777 listening and a full nmap scan shows no open port

lxc list c1copy
±-------±--------±---------------------±-----±-----------±----------+
| NAME | STATE | IPV4 | IPV6 | TYPE | SNAPSHOTS |
±-------±--------±---------------------±-----±-----------±----------+
| c1copy | RUNNING | 10.250.231.11 (eth0) | | PERSISTENT | 863 |
±-------±--------±---------------------±-----±-----------±----------+`

But worse still left in this state soon the whole ip stack will go wrong: example; all file mounts break and then only pings to “some ip address”- one core router but not the other!, Name service resolution fails, SSH etc…

to fix this:

Ip l s ens160 down && ip link set ens160 up

restores the all the connections until about 3-4 hours? then it will go wrong again. Removing the c1copyProxy removes the networking failure from happening.

Am i doing this right? what do you need to help trouble shoot this?
Thankyou.

2

Hi!

It works for me with the following.

First, look for an available IP address for the container. 10.10.10.199 looks good.

$ ping -c 1 10.10.10.199
PING 10.10.10.199 (10.10.10.199) 56(84) bytes of data.

--- 10.10.10.199 ping statistics ---
1 packets transmitted, 0 received, 100% packet loss, time 0ms

Run the following on the host. The IP address of the host is important to get right. In addition, when you give a static IP address to the container, you need to restart the container in order to switch the IP address.

$ lxc launch ubuntu:18.04 truenat
Creating truenat
Starting truenat
$ lxc config device override truenat eth0 ipv4.address=10.10.10.199
Device eth0 overridden for truenat
$ lxc restart truenat
$ lxc config device add truenat port80 proxy listen=tcp:192.168.1.100:80 connect=tcp:10.10.10.199:80 nat=true
Device port80 added to truenat
$

Run the following in the container.

$ lxc exec truenat -- apt update
$ lxc exec truenat -- apt install -y nginx

Now, you can access the Web server in the container using the IP address of the host. And the container should be able to get the real IP address of the client, not that of the proxy.

$ lxc exec truenat -- tail -1 /var/log/nginx/access.log
192.168.1.80 - - [17/Sep/2019:18:17:06 +0000] "GET / HTTP/1.1" 200 396 "-" "Mozilla/5.0 (Android 8.0; Mobile; rv:67.0) Gecko/67.0 Firefox/67.0"

All in all, I think that you did not restart the container after you set the static IP.
I do not know what could be the issue with the other networking issues that you have been facing. In terms of changes of LXD to your system, it just adds those iptables` rules.

3

Thanks for you reply, nice a clear.

I have restarted the whole server.
I have HAproxy running on the host OS would this cause a conflict?
I have two nics, I will remove one.

I am also going to build another host.

4

It would appear that LXD can only use deal with one NIC on the host, or bad things happen.

Its a big shame that the device ip address needs to be manually over ridden to another address, this means then that if i want to have all containers have a proxy device then i might as well turn DHCP off and manage all the addressing myself ! or is there a better way?