No DHCP request on CentOS7 or RedHat 7

LXC
1

LXC Focal Fossa container (nsa2) no DHCP Request. Only seeing this on CentOS 7 and RedHat 7 (on CentOS 8 this problem is not occurring).

[ubuntu@c7sv1 ~]$ sudo lxc-ls -f
[sudo] password for ubuntu:
NAME STATE AUTOSTART GROUPS IPV4 IPV6 UNPRIVILEGED
afns1 RUNNING 0 - 10.209.53.2, 172.29.108.2 - false
afns1-base STOPPED 0 - - - false
nsa2 STOPPED 0 - - - false
[ubuntu@c7sv1 ~]$ sudo lxc-start -n nsa2
[ubuntu@c7sv1 ~]$ sudo lxc-ls -f
NAME STATE AUTOSTART GROUPS IPV4 IPV6 UNPRIVILEGED
afns1 RUNNING 0 - 10.209.53.2, 172.29.108.2 - false
afns1-base STOPPED 0 - - - false
nsa2 RUNNING 0 - - - false
[ubuntu@c7sv1 ~]$ sudo lxc-stop -n nsa2
[ubuntu@c7sv1 ~]$ sudo lxc-start -n nsa2
[ubuntu@c7sv1 ~]$ sudo lxc-ls -f
NAME STATE AUTOSTART GROUPS IPV4 IPV6 UNPRIVILEGED
afns1 RUNNING 0 - 10.209.53.2, 172.29.108.2 - false
afns1-base STOPPED 0 - - - false
nsa2 RUNNING 0 - - - false
[ubuntu@c7sv1 ~]$ sudo lxc-attach -n nsa2 – dhclient
[ubuntu@c7sv1 ~]$ sudo lxc-ls -f
NAME STATE AUTOSTART GROUPS IPV4 IPV6 UNPRIVILEGED
afns1 RUNNING 0 - 10.209.53.2, 172.29.108.2 - false
afns1-base STOPPED 0 - - - false
nsa2 RUNNING 0 - 192.168.122.118 - false
[ubuntu@c7sv1 ~]$

Monitoring with tcpdump there is NO activity until the manual “dhclient” is fed to container:

[ubuntu@c7sv1 ~]$ sudo tcpdump -i any port 67 and port 68
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
li> stening on any, link-type LINUX_SLL (Linux cooked), capture size 262144 bytes

08:32:19.042709 IP 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from 00:16:3e:16:f7:5e (oui Unknown), length 300
08:32:19.042709 IP 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from 00:16:3e:16:f7:5e (oui Unknown), length 300
08:32:19.044613 IP c7sv1.bootps > 192.168.122.118.bootpc: BOOTP/DHCP, Reply, length 300
08:32:19.044619 IP c7sv1.bootps > 192.168.122.118.bootpc: BOOTP/DHCP, Reply, length 300

After the manual dhclient is fed from lxc-attach, now the container has an IP from virbr0:

NAME STATE AUTOSTART GROUPS IPV4 IPV6 UNPRIVILEGED
afns1 RUNNING 0 - 10.209.53.2, 172.29.108.2 - false
afns1-base STOPPED 0 - - - false
nsa2 RUNNING 0 - 192.168.122.118 - false
[ubuntu@c7sv1 ~]$

Thanks,

Gilbert

2

Well there is definitely something going on with Ubuntu Focal 20.04 LXC container DHCP request when running on CentOS 7.

I created in addition to focal, also xenial, bionic, and hirsute LXC containers, and all obtained DHCP automatically, EXCEPT for focal which requires a manual “dhclient” kick (sudo lxc-attach -n nsa1 – dhclient).

I tested all four Ubuntu releases on both virbr0 and lxdbr0 bridges, and the result was the same, so that tends to rule out some issue with either the bridge interface,or the DHCP server associated with the bridge, being the culprit.

These are vanilla LXC containers which have not been modified in any way. So the issue is something happening in the focal container itself when it interacts with CentOS 7 (i.e. focal LXC container obtains a DHCP address just fine on CentOS 8, Oracle Linux 7, Oracle Linux 8, etc).

For my requirement, any of these other Ubuntu releases (hirsuite, bionic, xenial) will be fine, so for CentOS 7 will just use a non-focal release, and that should meet my requirement.

I am fairly decent at getting to the bottom of these kinds of problems, but I confess this one has stumped me, so I will just workaround it by using a different Ubuntu release. The requirement has been met, but I’ve not identified a root cause so far.

My unfounded best guess atm is that it has something to do with the details of the particular implementation of systemd-networkd DHCP mechanism used in 20.04 which functions differently from the older dhclient mechanism, the clue being that the manual old-school dhclient step does work for 20.04 DHCP request on CentOS 7 host.

As noted previously, using tcpdump shows that the focal container is NOT issuing a DHCPREQUEST on CentOS 7 host until dhclient is run manually. For the other releases (bionic, xenial and hirsute) tcpdump detects the DHCPREQUEST traffic normally as expected.

[ubuntu@c7sv1 anylinux]$ sudo lxc-ls -f
NAME STATE AUTOSTART GROUPS IPV4 IPV6 UNPRIVILEGED
(Focal) nsa1 RUNNING 0 - - - false
(Hirsute) nsa2 RUNNING 0 - 192.168.122.123 - false
(Bionic) nsa3 RUNNING 0 - 192.168.122.176 - false
(Xenial) nsa4 RUNNING 0 - 192.168.122.71 - false

3

It’s indeed very very odd that focal would fail but hirsute succeed…
Anything useful in the systemd journal regarding networkd?
Or maybe something useful in the networkctl output?

4

Thanks for your reply! Update: Same issue occurring on RedHat 7 as well.

Environment: CentOS 7 with LXC 3.0.4 containers starting up on virbr0.
[ubuntu@c7sv1 ~]$ cat /etc/centos-release
CentOS Linux release 7.9.2009 (Core)
[ubuntu@c7sv1 ~]$ uname -a
Linux c7sv1 3.10.0-1160.42.2.el7.x86_64 #1 SMP Tue Sep 7 14:49:57 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

Containers just installed 100% vanilla.

Fossa:

root@nsa1:/home/ubuntu# cat /etc/lsb-release
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=20.04
DISTRIB_CODENAME=focal
DISTRIB_DESCRIPTION=“Ubuntu 20.04.3 LTS”
root@nsa1:/home/ubuntu# uname -a
Linux nsa1 3.10.0-1160.42.2.el7.x86_64 #1 SMP Tue Sep 7 14:49:57 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
root@nsa1:/home/ubuntu#

root@nsa1:/home/ubuntu# journalctl -b -u systemd-networkd
– Logs begin at Thu 2021-09-30 04:25:16 UTC, end at Fri 2021-10-01 01:11:29 UTC. –
Oct 01 01:08:02 nsa1 systemd[1]: Starting Network Service…
Oct 01 01:08:02 nsa1 systemd-networkd[56]: eth0: IPv6 successfully enabled
Oct 01 01:08:02 nsa1 systemd-networkd[56]: Could not enumerate nexthop: Invalid argument
Oct 01 01:08:02 nsa1 systemd[1]: systemd-networkd.service: Main process exited, code=exited, status=1/FAILURE
Oct 01 01:08:02 nsa1 systemd[1]: systemd-networkd.service: Failed with result ‘exit-code’.
Oct 01 01:08:02 nsa1 systemd[1]: Failed to start Network Service.
Oct 01 01:08:02 nsa1 systemd[1]: systemd-networkd.service: Scheduled restart job, restart counter is at 1.
Oct 01 01:08:02 nsa1 systemd[1]: Stopped Network Service.
Oct 01 01:08:02 nsa1 systemd[1]: Starting Network Service…
Oct 01 01:08:02 nsa1 systemd-networkd[62]: eth0: IPv6 successfully enabled
Oct 01 01:08:02 nsa1 systemd-networkd[62]: Could not enumerate nexthop: Invalid argument
Oct 01 01:08:02 nsa1 systemd[1]: systemd-networkd.service: Main process exited, code=exited, status=1/FAILURE
Oct 01 01:08:02 nsa1 systemd[1]: systemd-networkd.service: Failed with result ‘exit-code’.
Oct 01 01:08:02 nsa1 systemd[1]: Failed to start Network Service.
Oct 01 01:08:02 nsa1 systemd[1]: systemd-networkd.service: Scheduled restart job, restart counter is at 2.
Oct 01 01:08:02 nsa1 systemd[1]: Stopped Network Service.
Oct 01 01:08:02 nsa1 systemd[1]: Starting Network Service…
Oct 01 01:08:02 nsa1 systemd-networkd[65]: eth0: IPv6 successfully enabled
Oct 01 01:08:02 nsa1 systemd-networkd[65]: Could not enumerate nexthop: Invalid argument
Oct 01 01:08:02 nsa1 systemd[1]: systemd-networkd.service: Main process exited, code=exited, status=1/FAILURE
Oct 01 01:08:02 nsa1 systemd[1]: systemd-networkd.service: Failed with result ‘exit-code’.
Oct 01 01:08:02 nsa1 systemd[1]: Failed to start Network Service.
Oct 01 01:08:02 nsa1 systemd[1]: systemd-networkd.service: Scheduled restart job, restart counter is at 3.
Oct 01 01:08:02 nsa1 systemd[1]: Stopped Network Service.
Oct 01 01:08:02 nsa1 systemd[1]: Starting Network Service…
Oct 01 01:08:02 nsa1 systemd-networkd[70]: eth0: IPv6 successfully enabled
Oct 01 01:08:02 nsa1 systemd-networkd[70]: Could not enumerate nexthop: Invalid argument
Oct 01 01:08:02 nsa1 systemd[1]: systemd-networkd.service: Main process exited, code=exited, status=1/FAILURE
Oct 01 01:08:02 nsa1 systemd[1]: systemd-networkd.service: Failed with result ‘exit-code’.
Oct 01 01:08:02 nsa1 systemd[1]: Failed to start Network Service.
Oct 01 01:08:02 nsa1 systemd[1]: systemd-networkd.service: Scheduled restart job, restart counter is at 4.
Oct 01 01:08:02 nsa1 systemd[1]: Stopped Network Service.
Oct 01 01:08:02 nsa1 systemd[1]: Starting Network Service…
Oct 01 01:08:02 nsa1 systemd-networkd[79]: eth0: IPv6 successfully enabled
Oct 01 01:08:02 nsa1 systemd-networkd[79]: Could not enumerate nexthop: Invalid argument
Oct 01 01:08:02 nsa1 systemd[1]: systemd-networkd.service: Main process exited, code=exited, status=1/FAILURE
Oct 01 01:08:02 nsa1 systemd[1]: systemd-networkd.service: Failed with result ‘exit-code’.
Oct 01 01:08:02 nsa1 systemd[1]: Failed to start Network Service.
Oct 01 01:08:02 nsa1 systemd[1]: systemd-networkd.service: Scheduled restart job, restart counter is at 5.
Oct 01 01:08:02 nsa1 systemd[1]: Stopped Network Service.
Oct 01 01:08:02 nsa1 systemd[1]: systemd-networkd.service: Start request repeated too quickly.
Oct 01 01:08:02 nsa1 systemd[1]: systemd-networkd.service: Failed with result ‘exit-code’.
Oct 01 01:08:02 nsa1 systemd[1]: Failed to start Network Service.

Xenial, Bionic, Hirsute:

root@nsa4:/home/ubuntu# journalctl -b -u systemd-networkd
– Journal begins at Thu 2021-09-30 04:25:58 UTC, ends at Fri 2021-10-01 01:17:01 UTC. –
Oct 01 01:07:28 nsa4 systemd[1]: Starting Network Service…
Oct 01 01:07:28 nsa4 systemd-networkd[53]: Enumeration completed
Oct 01 01:07:28 nsa4 systemd[1]: Started Network Service.
Oct 01 01:07:29 nsa4 systemd-networkd[53]: eth0: Gained IPv6LL
Oct 01 01:07:32 nsa4 systemd-networkd[53]: eth0: DHCPv4 address 192.168.122.71/24 via 192.168.122.1

As mentioned I’ve a workaround using Hirsute for the DNS/DHCP container for CentOS 7 deployments so I will have to circle back to this a bit later to get to the bottom of what is going on with systemd-networkd that is causing it to fail out of the box Fossa only & CentOS 7/RedHat 7 only.

5

Update: Same issue is occurring on RedHat 7 as well. Same workaround used successfully.