Containers do not acquire IPv4 address after being transferred to new host

Hello dear LXC people,

I did transfer a bunch of containers from an Ubuntu 18.04 LXD version 3.0.3 (installed from repos) to an Ubuntu 20.04 LXD version 4.0.6 (installed with snap).

Three privileged containers now fail to acquire an IPv4 address on the new LXD host. All the containers are set up with the standard profile, so get the “same” eth0 device. Their networking works just fine on the previous LXD host. All non-privileged containers work perfectly on the LXD host.

Where do I need to start looking for the source of the glitch? I’m not a networking expert, and didn’t have these issues before after transferring privileged containers.

I’d be grateful for any help and suggestions.

The usual suspects for that kind of issues are:

  • Docker firewall (you can check with iptables -L -n -v)
  • UFW rules (same check as above, possibly nft list ruleset too)
  • DNS port conflict (netstat -lnp)

Thank you Stéphane.

The machine running LXD is a fresh install of Ubuntu 20.04 for the sole purpose of running containers, so I hope that there isn’t too much fuss happening on the host level.

I have tried changing the privileged flag to false for the offending containers, as there were no issues with non-privileged containers, and as expected their file systems got remapped, but there was no effect on ipv4 connectivity.

iptables returns no rules being in place:

sudo iptables -L -n -v
Chain INPUT (policy ACCEPT 726K packets, 84M bytes)
 pkts bytes target     prot opt in     out     source               destination         

Chain FORWARD (policy ACCEPT 60875 packets, 19M bytes)
 pkts bytes target     prot opt in     out     source               destination         

Chain OUTPUT (policy ACCEPT 697K packets, 7909M bytes)
 pkts bytes target     prot opt in     out     source               destination         

ufw is installed, but inactive.

sudo nft list ruleset
table inet lxd {
	chain pstrt.lxdbr0 {
		type nat hook postrouting priority srcnat; policy accept;
		@nh,96,24 704899 @nh,128,24 != 704899 masquerade
	}

	chain fwd.lxdbr0 {
		type filter hook forward priority filter; policy accept;
		ip version 4 oifname "lxdbr0" accept
		ip version 4 iifname "lxdbr0" accept
	}

	chain in.lxdbr0 {
		type filter hook input priority filter; policy accept;
		iifname "lxdbr0" tcp dport 53 accept
		iifname "lxdbr0" udp dport 53 accept
		iifname "lxdbr0" udp dport 67 accept
	}

	chain out.lxdbr0 {
		type filter hook output priority filter; policy accept;
		oifname "lxdbr0" tcp sport 53 accept
		oifname "lxdbr0" udp sport 53 accept
		oifname "lxdbr0" udp sport 67 accept
	}
}

For DNS, I did set up resolved to be able to use container names on the host.

In the netstat output i replaced the names of containers that receive an ipv4 addres with good-n and the ones that do not with bad-n:

sudo netstat -lnp
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name    
tcp        0      0 10.193.131.1:53         0.0.0.0:*               LISTEN      39594/dnsmasq       
tcp        0      0 127.0.0.53:53           0.0.0.0:*               LISTEN      947/systemd-resolve 
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      1247/sshd: /usr/sbi 
tcp6       0      0 fe80::d872:8cff:fe88:53 :::*                    LISTEN      39594/dnsmasq       
tcp6       0      0 :::22                   :::*                    LISTEN      1247/sshd: /usr/sbi 
tcp6       0      0 :::8443                 :::*                    LISTEN      1563/lxd            
udp        0      0 10.193.131.1:53         0.0.0.0:*                           39594/dnsmasq       
udp        0      0 127.0.0.53:53           0.0.0.0:*                           947/systemd-resolve 
udp        0      0 0.0.0.0:67              0.0.0.0:*                           39594/dnsmasq       
udp        0      0 10.150.0.7:68           0.0.0.0:*                           944/systemd-network 
udp        0      0 127.0.0.1:323           0.0.0.0:*                           995/chronyd         
udp6       0      0 fe80::d872:8cff:fe88:53 :::*                                39594/dnsmasq       
udp6       0      0 ::1:323                 :::*                                995/chronyd         
raw6       0      0 :::58                   :::*                    7           944/systemd-network 
raw6       0      0 :::58                   :::*                    7           944/systemd-network 
Active UNIX domain sockets (only servers)
Proto RefCnt Flags       Type       State         I-Node   PID/Program name     Path
unix  2      [ ACC ]     STREAM     LISTENING     27672    1563/lxd             /var/snap/lxd/common/lxd/devlxd/sock
unix  2      [ ACC ]     STREAM     LISTENING     4655965  1469234/systemd      /run/user/1000/systemd/private
unix  2      [ ACC ]     STREAM     LISTENING     4655970  1469234/systemd      /run/user/1000/bus
unix  2      [ ACC ]     STREAM     LISTENING     4655971  1469234/systemd      /run/user/1000/gnupg/S.dirmngr
unix  2      [ ACC ]     STREAM     LISTENING     4655972  1469234/systemd      /run/user/1000/gnupg/S.gpg-agent.browser
unix  2      [ ACC ]     STREAM     LISTENING     4655973  1469234/systemd      /run/user/1000/gnupg/S.gpg-agent.extra
unix  2      [ ACC ]     STREAM     LISTENING     2407     1/init               @/org/kernel/linux/storage/multipathd
unix  2      [ ACC ]     STREAM     LISTENING     4655974  1469234/systemd      /run/user/1000/gnupg/S.gpg-agent.ssh
unix  2      [ ACC ]     STREAM     LISTENING     4655975  1469234/systemd      /run/user/1000/gnupg/S.gpg-agent
unix  2      [ ACC ]     STREAM     LISTENING     4655976  1469234/systemd      /run/user/1000/pk-debconf-socket
unix  2      [ ACC ]     STREAM     LISTENING     4655978  1469234/systemd      /run/user/1000/snapd-session-agent.socket
unix  2      [ ACC ]     STREAM     LISTENING     30314    3899/[lxc monitor]   @/var/snap/lxd/common/lxd/containers/good-1/command
unix  2      [ ACC ]     STREAM     LISTENING     20997    1/init               /run/dbus/system_bus_socket
unix  2      [ ACC ]     STREAM     LISTENING     19962    1/init               /run/snapd.socket
unix  2      [ ACC ]     STREAM     LISTENING     19964    1/init               /run/snapd-snap.socket
unix  2      [ ACC ]     STREAM     LISTENING     19966    1/init               /run/uuidd/request
unix  2      [ ACC ]     STREAM     LISTENING     33216    5029/[lxc monitor]   @/var/snap/lxd/common/lxd/containers/good-2/command
unix  2      [ ACC ]     STREAM     LISTENING     2394     1/init               /run/systemd/private
unix  2      [ ACC ]     STREAM     LISTENING     2396     1/init               /run/systemd/userdb/io.systemd.DynamicUser
unix  2      [ ACC ]     STREAM     LISTENING     3289299  583201/[lxc monitor  @/var/snap/lxd/common/lxd/containers/good-3/command
unix  2      [ ACC ]     STREAM     LISTENING     19960    1/init               /var/snap/lxd/common/lxd/unix.socket
unix  2      [ ACC ]     STREAM     LISTENING     2405     1/init               /run/lvm/lvmpolld.socket
unix  2      [ ACC ]     STREAM     LISTENING     2410     1/init               /run/systemd/fsck.progress
unix  2      [ ACC ]     STREAM     LISTENING     3319120  594706/[lxc monitor  @/var/snap/lxd/common/lxd/containers/good-4/command
unix  2      [ ACC ]     STREAM     LISTENING     2420     1/init               /run/systemd/journal/stdout
unix  2      [ ACC ]     SEQPACKET  LISTENING     2425     1/init               /run/udev/control
unix  2      [ ACC ]     STREAM     LISTENING     15528    186/systemd-journal  /run/systemd/journal/io.systemd.journal
unix  2      [ ACC ]     STREAM     LISTENING     3401324  639599/[lxc monitor  @/var/snap/lxd/common/lxd/containers/good-5/command
unix  2      [ ACC ]     STREAM     LISTENING     4692658  1490069/[lxc monito  @/var/snap/lxd/common/lxd/containers/bad-1/command
unix  2      [ ACC ]     STREAM     LISTENING     4697285  1492963/[lxc monito  @/var/snap/lxd/common/lxd/containers/bad-2/command
unix  2      [ ACC ]     STREAM     LISTENING     26845    1705/[lxc monitor]   @/var/snap/lxd/common/lxd/containers/good-6/command
unix  2      [ ACC ]     STREAM     LISTENING     35279    7831/[lxc monitor]   @/var/snap/lxd/common/lxd/containers/bad-3/command
unix  2      [ ACC ]     STREAM     LISTENING     19959    1/init               @ISCSIADM_ABSTRACT_NAMESPACE
unix  2      [ ACC ]     SEQPACKET  LISTENING     29179    1563/lxd             /var/snap/lxd/common/lxd/seccomp.socket
unix  2      [ ACC ]     STREAM     LISTENING     3350487  609875/[lxc monitor  @/var/snap/lxd/common/lxd/containers/good-7/command
unix  2      [ ACC ]     STREAM     LISTENING     26648    1563/lxd             @00010

From the inside of a “bad” container, with eth0 sporting an ipv6 address:

ifconfig
eth0      Link encap:Ethernet  HWaddr 00:16:3e:1c:c4:52  
          inet6 addr: fe80::216:3eff:fe1c:c452/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:108 errors:0 dropped:0 overruns:0 frame:0
          TX packets:28 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:17616 (17.6 KB)  TX bytes:6680 (6.6 KB)

lo        Link encap:Local Loopback  
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:65536  Metric:1
          RX packets:4 errors:0 dropped:0 overruns:0 frame:0
          TX packets:4 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:308 (308.0 B)  TX bytes:308 (308.0 B)

The container seems to have the device configured correctly:

lxc config show collective-access --expanded
architecture: x86_64
config:
  image.architecture: amd64
  image.description: ubuntu 12.04 LTS amd64 (release) (20170502)
  image.label: release
  image.os: ubuntu
  image.release: precise
  image.serial: "20170502"
  image.type: root.tar.xz
  image.version: "12.04"
  security.privileged: "false"
  volatile.base_image: be4aa8e56eab681fac6553b48ce19d7f34833accc2c8ae65f140a603b8369a1d
  volatile.eth0.host_name: veth91e363a6
  volatile.eth0.hwaddr: 00:16:3e:1c:c4:52
  volatile.idmap.base: "0"
  volatile.idmap.current: '[{"Isuid":true,"Isgid":false,"Hostid":1000000,"Nsid":0,"Maprange":1000000000},{"Isuid":false,"Isgid":true,"Hostid":1000000,"Nsid":0,"Maprange":1000000000}]'
  volatile.idmap.next: '[{"Isuid":true,"Isgid":false,"Hostid":1000000,"Nsid":0,"Maprange":1000000000},{"Isuid":false,"Isgid":true,"Hostid":1000000,"Nsid":0,"Maprange":1000000000}]'
  volatile.last_state.idmap: '[{"Isuid":true,"Isgid":false,"Hostid":1000000,"Nsid":0,"Maprange":1000000000},{"Isuid":false,"Isgid":true,"Hostid":1000000,"Nsid":0,"Maprange":1000000000}]'
  volatile.last_state.power: RUNNING
  volatile.uuid: d5451aec-2e03-4b75-8bb0-3892d4887447
devices:
  eth0:
    name: eth0
    network: lxdbr0
    type: nic
  root:
    path: /
    pool: zfspool
    type: disk
ephemeral: false
profiles:
- default
stateful: false
description: ""

The managed network config looks like below,:

lxc network show lxdbr0
config:
  ipv4.address: 10.193.131.1/24
  ipv4.nat: "true"
  ipv6.address: none
  raw.dnsmasq: |-
    auth-zone=lxd
    dns-loop-detect
description: ""
name: lxdbr0
type: bridge
used_by:
- /1.0/instances/good-1
- /1.0/instances/bad-1
- /1.0/instances/bad-2
- /1.0/instances/good-2
- /1.0/instances/good-3
- /1.0/instances/good-4
- /1.0/instances/good-5
- /1.0/instances/good-6
- /1.0/instances/good-7
- /1.0/instances/good-8
- /1.0/profiles/default
managed: true
status: Created
locations:
- none

Yeah, that’s all looking correct and clean to me. So that suggests some kind of issue with the containers instead.

Can you run systemctl --failed in those containers as well as ps fauxww?

This is from inside a container with issues:

sudo ps fauxww
sudo: unable to resolve host bad-1
USER         PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root        3728  0.0  0.0  41228  1880 pts/0    Ss   14:41   0:00 su -l
root        3729  0.0  0.0  22112  5000 pts/0    S    14:41   0:00  \_ -su
root        3788  0.0  0.0  41920  2172 pts/0    S+   14:42   0:00      \_ sudo ps fauxww
root        3792  0.0  0.0  16864  1668 pts/0    R+   14:42   0:00          \_ ps fauxww
root           1  0.0  0.0  24336  2376 ?        Ss   04:45   0:00 /sbin/init
root          35  0.0  0.0  25308  1816 ?        S    04:45   0:00 mountall --daemon
root         331  0.0  0.0  15204   196 ?        S    04:45   0:00 upstart-socket-bridge --daemon
root        2529  0.0  0.0  17248   164 ?        S    04:45   0:00 upstart-udev-bridge --daemon
root        2531  0.0  0.0  21352  1600 ?        Ss   04:45   0:00 /sbin/udevd --daemon
102         2571  0.0  0.0  23832  1052 ?        Ss   04:47   0:00 dbus-daemon --system --fork --activation=upstart
syslog      2572  0.0  0.0 180004  2112 ?        Sl   04:47   0:02 rsyslogd -c5
daemon      2691  0.0  0.0  16924   156 ?        Ss   04:49   0:00 atd
root        2693  0.0  0.0  19128  1588 ?        Ss   04:49   0:00 cron
mysql       2705  0.0  1.3 575312 106072 ?       Ssl  04:49   0:07 /usr/sbin/mysqld
root        2708  0.0  0.0  15996  1384 ?        Ss   04:49   0:03 /usr/sbin/irqbalance
whoopsie    2743  0.0  0.0 122088  6280 ?        Ss   04:49   0:00 whoopsie
root        2763  0.0  0.2 320684 17840 ?        Ss   04:49   0:01 /usr/sbin/apache2 -k start
www-data    3164  0.0  0.1 320708  8948 ?        S    06:29   0:00  \_ /usr/sbin/apache2 -k start
www-data    3165  0.0  0.1 320708  8948 ?        S    06:29   0:00  \_ /usr/sbin/apache2 -k start
www-data    3167  0.0  0.1 320708  8948 ?        S    06:29   0:00  \_ /usr/sbin/apache2 -k start
www-data    3168  0.0  0.1 320708  8948 ?        S    06:29   0:00  \_ /usr/sbin/apache2 -k start
www-data    3170  0.0  0.1 320708  8948 ?        S    06:29   0:00  \_ /usr/sbin/apache2 -k start
root        2785  0.0  0.0   4344  1252 ?        Ss   04:49   0:00 acpid -c /etc/acpi/events -s /var/run/acpid.socket
root        2903  0.0  0.0   7292  1696 ?        Ss   04:50   0:00 dhclient3 -e IF_METRIC=100 -pf /var/run/dhclient.eth0.pid -lf /var/lib/dhcp/dhclient.eth0.leases -1 eth0
root        2926  0.0  0.0  50048  3492 ?        Ss   04:50   0:00 /usr/sbin/sshd -D

None of the offending containers have systemd installed. They’re running older versions of Ubuntu or Debian. There are other containers with the same operating systems that work just fine.

Maybe the following can be helpful? cloud-init-nonet apparently couldn’t find a network device:

lxc console --show-log bad-1
Console log:

/bin/sh: 1: lxcfs: not found
mountall: mount /proc/cpuinfo [100] terminated with status 127
/bin/sh: 1: lxcfs: not found
mountall: mount /proc/diskstats [106] terminated with status 127
/bin/sh: 1: lxcfs: not found
mountall: mount /proc/loadavg [113] terminated with status 127
/bin/sh: 1: lxcfs: not found
mountall: mount /proc/meminfo [121] terminated with status 127
/bin/sh: 1: lxcfs: not found
mountall: mount /proc/stat [129] terminated with status 127
/bin/sh: 1: lxcfs: not found
mountall: mount /proc/swaps [134] terminated with status 127
/bin/sh: 1: lxcfs: not found
mountall: mount /proc/uptime [141] terminated with status 127
/bin/sh: 1: lxcfs: not found
mountall: mount /sys/devices/system/cpu/online [147] terminated with status 127
mountall: Event failed
cloud-init-container: emitted ifup for eth0
cloud-init start-local running: Sun, 06 Jun 2021 08:45:27 +0000. up 0.36 seconds
no instance data found in start-local
mountall: Disconnected from Plymouth
cloud-init-nonet waiting 120 seconds for a network device.
cloud-init-nonet gave up waiting for a network device.
ci-info: lo    : 1 127.0.0.1       255.0.0.0       .
ci-info: eth0  : 1 .               .               00:16:3e:1c:c4:52
TERM environment variable not set.
lxc
 * Not starting AppArmor in container
   ...done.
landscape-client is not configured, please run landscape-config.
 * Starting web server apache2
apache2: apr_sockaddr_info_get() failed for collective-access
apache2: Could not reliably determine the server's fully qualified domain name, using 127.0.0.1 for ServerName
   ...done.

What’s the value of lxc info | grep firewall: on the new server?

I wonder if those old containers somehow rely on the old xtables mangle rule to fill UDP checksums which no longer exists when using nft.

1 Like

This is the result on the new lxc host:

ubuntu@lxc-host-r2:~$ lxc info | grep firewall:
  firewall: nftables

The old host returns no firewall key from lxc info.

Indeed there seems to be an UDP checksum issue:

ubuntu@lxc-host-r2:~$ lxc shell bad-1
root@bad-1:~# dhclient -v eth0
Internet Systems Consortium DHCP Client 4.1-ESV-R4
Copyright 2004-2011 Internet Systems Consortium.
All rights reserved.
For info, please visit https://www.isc.org/software/dhcp/

Listening on LPF/eth0/00:16:3e:1c:c4:52
Sending on   LPF/eth0/00:16:3e:1c:c4:52
Sending on   Socket/fallback
DHCPDISCOVER on eth0 to 255.255.255.255 port 67 interval 3
DHCPDISCOVER on eth0 to 255.255.255.255 port 67 interval 5
DHCPDISCOVER on eth0 to 255.255.255.255 port 67 interval 7
5 bad udp checksums in 5 packets
DHCPDISCOVER on eth0 to 255.255.255.255 port 67 interval 12
DHCPDISCOVER on eth0 to 255.255.255.255 port 67 interval 16
5 bad udp checksums in 5 packets

OK so in that case you should get LXD to use the legacy xtables driver (iptables, ip6tables and ebtables) as this supports fixing checksums from older DHCP clients.

To do this you need to make sure you have iptables, ip6tables and ebtables installed and that they are not the nftable shim versions (running each command with --version and checking they don’t contain mentions of “nft” should do). See nftables - Debian Wiki

Then you need to clear your nftables ruleset using sudo nft flush ruleset (assuming you’re happy to lose those rules).

Then indicate to LXD that xtables is in use by adding a 1 or more iptables rules, e.g.

sudo iptables -A INPUT

Then reload LXD:

sudo systemctl reload snap.lxd.daemon

Thank you Thomas!

I now did purge nftables from the host system, and installed the legacy versions:

ubuntu@lxc-host-r2:~$ sudo ebtables --version
ebtables v2.0.11 (legacy) (December 2011)
ubuntu@lxc-host-r2:~$ sudo arptables --version
arptables v0.0.5
ubuntu@lxc-host-r2:~$ sudo iptables --version
iptables v1.8.4 (legacy)
ubuntu@lxc-host-r2:~$ sudo ip6tables --version
ip6tables v1.8.4 (legacy)

However, the containers still cannot obtain an ipv4 address and give an UDP checksum error:

ubuntu@lxc-host-r2:~$ lxc shell bad-1
root@collective-access:~# dhclient -v eth0
Internet Systems Consortium DHCP Client 4.1-ESV-R4
Copyright 2004-2011 Internet Systems Consortium.
All rights reserved.
For info, please visit https://www.isc.org/software/dhcp/

Listening on LPF/eth0/00:16:3e:1c:c4:52
Sending on   LPF/eth0/00:16:3e:1c:c4:52
Sending on   Socket/fallback
DHCPDISCOVER on eth0 to 255.255.255.255 port 67 interval 3
DHCPDISCOVER on eth0 to 255.255.255.255 port 67 interval 8
DHCPDISCOVER on eth0 to 255.255.255.255 port 67 interval 15
5 bad udp checksums in 5 packets

Does the snap version of LXD use the system’s iptables, or does it come bundled with its own?

What does lxc info | grep firewall: show?

Yes the snap package comes with xtables and nftables inside it, removing the nftables package on your host and ensuring the legacy xtables tools are installed are just to prevent another application from populating the nftables ruleset externall (which LXD will then take as a signal to use its internal nftables itself).

If you reinstall nftables what does sudo nft list ruleset and sudo iptables-save show now?

Oh it indeed looks like the firewall value is still wrong:

ubuntu@lxc-host-r2:~$  lxc info | grep firewall:
  firewall: nftables

Can this be set to something else?

Yes, you need to clear nftables ruleset and ensure there’s at least one iptables rule active.

See Containers do not acquire IPv4 address after being transferred to new host - #8 by tomp

OK, I verified that I did all of these things:

  1. Cleared the nft ruleset, purged nf_tables from the host, restarted the host.
  2. Created a dummy iptables rule:
    $ sudo iptables -A INPUT
    $ sudo iptables --list
    Chain INPUT (policy ACCEPT)
    target     prot opt source               destination         
              all  --  anywhere             anywhere            
              all  --  anywhere             anywhere            
    
    Chain FORWARD (policy ACCEPT)
    target     prot opt source               destination         
    
    Chain OUTPUT (policy ACCEPT)
    target     prot opt source               destination
    
  3. Reloaded LXD via sudo systemctl reload snap.lxd.daemon

However, LXD still lists nftables as the firewall:

$ lxc info | grep firewall:
  firewall: nftables

It was required to fully restart LXD, using sudo systemctl restart snap.lxd.daemon for it to recognize the new setup:

ubuntu@lxc-host-r2:~$ sudo systemctl restart snap.lxd.daemon
ubuntu@lxc-host-r2:~$ lxc info | grep firewall:
  firewall: xtables

All legacy containers now acquire ipv4 addresses.

:sunflower: Big win!! Thank you both so much Stéphane and Thomas! :sunflower:

Attention: iptables rules are reset after every restart of the host. There is a package for Ubuntu that helps to set up the rules on every system start. Here is how I did it, as the root user:

iptables -A INPUT  # set up iptables rule as discussed above
apt install iptables-persistent

The interactive package installer prompts if the current settings should be saved, which should be fine.

If you want to change the saved settings later, use

iptables-save > /etc/iptables/rules.v4

It is amazing that LXD is able to handle such a case of legacy DHCP clients, of which I wasn’t even aware could be an issue.

I was now wondering what the status of xtables is in general: apparently it is being replaced by nftables, and might at some point not be available anymore in mainstream Linux distributions. Would that impact the networking abilities of LXC containers in the long run?

It may impact some older containers indeed once distros default to nft and drop xtables. Those may need to switch to another DHCP client or even go with static addressing.

2 Likes