No networking in containers, missing dnsmasq? snap on debian

Hi,

I’ve been happily using the lxd snap on Debian buster for some time, with many containers.

I recently upgraded to Debian bullseye, and after a reboot, my containers didn’t have any networking anymore.

When I would lxc shell into one of the containers, I would see this:

WARNING: cgroup v2 is not fully supported yet, proceeding with partial confinement

I searched around and found some posts related to this and I found that if I added systemd.unified_cgroup_hierarchy=0 to my grub linux commandline, rebuild the grub config and then rebooted, that message went away.

However, I still do not have networking. I see that lxd network list shows me the bridge interface and it has an IP.

I noticed I could enter the container and manually set an IP on the interface, and set the default route, and then edit the resolv.conf and things worked. So it seems like the containers are not successfully doing DHCP.

I seem to recall seeing on the host a LXD-specific dnsmasq running, but I dont see that now.

What can I look for to find out why this dnsmasq isn’t being launched with my lxd?

Thanks for any suggestions! I’m happy to provide any debugging info I can.

Please provide output of the following on the host:

ip a
ip r
sudo ps aux | grep dnsmasq
sudo ss -ulpn
# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: enp0s25: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether d0:50:99:8e:a7:87 brd ff:ff:ff:ff:ff:ff
    inet 192.168.1.111/24 brd 192.168.1.255 scope global dynamic noprefixroute enp0s25
       valid_lft 42008sec preferred_lft 42008sec
    inet6 fe80::d250:99ff:fe8e:a787/64 scope link noprefixroute 
       valid_lft forever preferred_lft forever
3: lxdbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 00:16:3e:e2:8d:a0 brd ff:ff:ff:ff:ff:ff
    inet 10.155.80.1/24 scope global lxdbr0
       valid_lft forever preferred_lft forever
    inet6 fd42:fce7:f7ba:df2a::1/64 scope global 
       valid_lft forever preferred_lft forever
    inet6 fe80::216:3eff:fee2:8da0/64 scope link 
       valid_lft forever preferred_lft forever
7: veth3417b0c8@if6: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master lxdbr0 state UP group default qlen 1000
    link/ether de:6b:7f:45:1a:26 brd ff:ff:ff:ff:ff:ff link-netnsid 0
9: veth42575a99@if8: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master lxdbr0 state UP group default qlen 1000
    link/ether c2:02:bb:ad:a0:58 brd ff:ff:ff:ff:ff:ff link-netnsid 1
# ip r
default via 192.168.1.1 dev enp0s25 proto dhcp metric 100 
10.155.80.0/24 dev lxdbr0 proto kernel scope link src 10.155.80.1 
192.168.1.0/24 dev enp0s25 proto kernel scope link src 192.168.1.111 metric 100 
# ps aux |grep dnsmasq
dnsmasq    16512  0.0  0.0  19872  2156 ?        S    Apr13   0:00 /usr/sbin/dnsmasq -x /run/dnsmasq/dnsmasq.pid -u dnsmasq -r /run/dnsmasq/resolv.conf -7 /etc/dnsmasq.d,.dpkg-dist,.dpkg-old,.dpkg-new --local-service --trust-anchor=.,20326,8,2,e06d44b80b8f1d39a95c0b0d7c65d08458e880409bbc683457104237c7f8ec8d
root     1587752  0.0  0.0  11536   716 pts/13   S+   10:19   0:00 grep dnsmasq
# ss -ulpn
State                Recv-Q                Send-Q                                                     Local Address:Port                                Peer Address:Port               Process                                                                 
UNCONN               0                     0                                                                0.0.0.0:48869                                    0.0.0.0:*                   users:(("mosh-client",pid=12566,fd=4))                                 
UNCONN               0                     0                                                                0.0.0.0:53                                       0.0.0.0:*                   users:(("dnsmasq",pid=16512,fd=4))                                     
UNCONN               0                     0                                                                0.0.0.0:111                                      0.0.0.0:*                   users:(("rpcbind",pid=864,fd=5),("systemd",pid=1,fd=57))               
UNCONN               0                     0                                                            10.155.80.1:123                                      0.0.0.0:*                   users:(("ntpd",pid=1096,fd=25))                                        
UNCONN               0                     0                                                          192.168.1.111:123                                      0.0.0.0:*                   users:(("ntpd",pid=1096,fd=23))                                        
UNCONN               0                     0                                                              127.0.0.1:123                                      0.0.0.0:*                   users:(("ntpd",pid=1096,fd=18))                                        
UNCONN               0                     0                                                                0.0.0.0:123                                      0.0.0.0:*                   users:(("ntpd",pid=1096,fd=17))                                        
UNCONN               0                     0                                                                0.0.0.0:49705                                    0.0.0.0:*                   users:(("rpc.mountd",pid=1076,fd=8))                                   
UNCONN               0                     0                                                                0.0.0.0:631                                      0.0.0.0:*                   users:(("cups-browsed",pid=1174752,fd=7))                              
UNCONN               0                     0                                                                0.0.0.0:2049                                     0.0.0.0:*                                                                                          
UNCONN               0                     0                                                                0.0.0.0:44225                                    0.0.0.0:*                                                                                          
UNCONN               0                     0                                                                0.0.0.0:52577                                    0.0.0.0:*                   users:(("apt-cacher-ng",pid=1047,fd=13))                               
UNCONN               0                     0                                                                0.0.0.0:54681                                    0.0.0.0:*                   users:(("rpc.mountd",pid=1076,fd=12))                                  
UNCONN               0                     0                                                                0.0.0.0:39338                                    0.0.0.0:*                   users:(("rpc.mountd",pid=1076,fd=16))                                  
UNCONN               0                     0                                                                   [::]:53                                          [::]:*                   users:(("dnsmasq",pid=16512,fd=6))                                     
UNCONN               0                     0                                                                   [::]:111                                         [::]:*                   users:(("rpcbind",pid=864,fd=7),("systemd",pid=1,fd=60))               
UNCONN               0                     0                                      [fe80::216:3eff:fee2:8da0]%lxdbr0:123                                         [::]:*                   users:(("ntpd",pid=1096,fd=27))                                        
UNCONN               0                     0                                               [fd42:fce7:f7ba:df2a::1]:123                                         [::]:*                   users:(("ntpd",pid=1096,fd=26))                                        
UNCONN               0                     0                                    [fe80::d250:99ff:fe8e:a787]%enp0s25:123                                         [::]:*                   users:(("ntpd",pid=1096,fd=24))                                        
UNCONN               0                     0                                                                  [::1]:123                                         [::]:*                   users:(("ntpd",pid=1096,fd=19))                                        
UNCONN               0                     0                                                                   [::]:123                                         [::]:*                   users:(("ntpd",pid=1096,fd=16))                                        
UNCONN               0                     0                                                                   [::]:57838                                       [::]:*                   users:(("rpc.mountd",pid=1076,fd=18))                                  
UNCONN               0                     0                                                                   [::]:59132                                       [::]:*                   users:(("rpc.mountd",pid=1076,fd=14))                                  
UNCONN               0                     0                                                                   [::]:2049                                        [::]:*                                                                                          
UNCONN               0                     0                                                                   [::]:59915                                       [::]:*                                                                                          
UNCONN               0                     0                                                                   [::]:54387                                       [::]:*                   users:(("rpc.mountd",pid=1076,fd=10))                                  

You have another instance of dnsmasq that is listening on port 53 on all interfaces, this will prevent LXD’s dnsmasq from starting and listening only on lxdbr0 for DNS and DHCP.

Please disable the dnsmasq instance or reconfigure it to not listen on lxdbr0.

Indeed, this seems to be the problem. Thank you!!

1 Like