After an Ubuntu upgrade the network stopped working

NinoBYSA · January 13, 2021, 2:18pm

Hello, after an Ubuntu upgrade my containers don’t get an IP and the name resolution don’t work either (ping containername.lxd).

I’m running LXD 4.10, here’s the result of lxc network list:

+--------+----------+---------+-----------------+---------------------------+-------------+---------+
|  NAME  |   TYPE   | MANAGED |      IPV4       |           IPV6            | DESCRIPTION | USED BY |
+--------+----------+---------+-----------------+---------------------------+-------------+---------+
| enp1s0 | physical | NO      |                 |                           |             | 0       |
+--------+----------+---------+-----------------+---------------------------+-------------+---------+
| enp2s0 | physical | NO      |                 |                           |             | 0       |
+--------+----------+---------+-----------------+---------------------------+-------------+---------+
| lxdbr0 | bridge   | YES     | 10.206.203.1/24 | fd42:5588:e06d:ebd0::1/64 |             | 6       |
+--------+----------+---------+-----------------+---------------------------+-------------+---------+

I don’t know where to look up .

tomp · January 13, 2021, 2:30pm

The usual steps to resolve issues like this are as follows:

Confirm LXD’s dnsmasq is running and listening on your bridge’s IP (using sudo ps aux | grep dnsmasq and sudo ss -ulpn | grep dnsmasq and that nothing else is preventing LXD’s dnsmasq from listening.
If you have docker running on the host check its firewall rules aren’t blocking DHCP requests to lxdbr0. See Lxd and Docker Firewall Redux - How to deal with FORWARD policy set to drop
Check that you haven’t got anything else that could be adding firewall rules (using sudo iptables-save and/or sudo nft list ruleset) that could be blocking LXD’s dnsmasq from getting DHCP requests.

NinoBYSA · January 13, 2021, 2:36pm

dsnmasq isn’t running, ps aux | grep dnsmasq gives only the grep line.
Docker isn’t installed on the host.
I don’t know what to look for in iptables rules, there’s still the ones commented as “generated for LXD network lxdbr0”.

tomp · January 13, 2021, 2:38pm

You’ll most likely have something else listening on port 53 or DHCP ports that is preventing dnsmasq from starting.

Please show full output of sudo ss -ulpn

NinoBYSA · January 13, 2021, 2:42pm

Indeed, lsof -i :53 show systemd-resolve and bind.

ss -ulpn

State          Recv-Q          Send-Q                                       Local Address:Port                   Peer Address:Port         Process                                            
UNCONN         0               0                                             10.206.203.1:53                          0.0.0.0:*             users:(("named",pid=909,fd=144))                  
UNCONN         0               0                                             10.206.203.1:53                          0.0.0.0:*             users:(("named",pid=909,fd=148))                  
UNCONN         0               0                                             10.206.203.1:53                          0.0.0.0:*             users:(("named",pid=909,fd=147))                  
UNCONN         0               0                                             10.206.203.1:53                          0.0.0.0:*             users:(("named",pid=909,fd=146))                  
UNCONN         0               0                                             10.206.203.1:53                          0.0.0.0:*             users:(("named",pid=909,fd=143))                  
UNCONN         0               0                                             10.206.203.1:53                          0.0.0.0:*             users:(("named",pid=909,fd=145))                  
UNCONN         0               0                                             10.206.203.1:53                          0.0.0.0:*             users:(("named",pid=909,fd=142))                  
UNCONN         0               0                                             10.206.203.1:53                          0.0.0.0:*             users:(("named",pid=909,fd=141))                  
UNCONN         0               0                                            91.121.165.12:53                          0.0.0.0:*             users:(("named",pid=909,fd=110))                  
UNCONN         0               0                                            91.121.165.12:53                          0.0.0.0:*             users:(("named",pid=909,fd=108))                  
UNCONN         0               0                                            91.121.165.12:53                          0.0.0.0:*             users:(("named",pid=909,fd=115))                  
UNCONN         0               0                                            91.121.165.12:53                          0.0.0.0:*             users:(("named",pid=909,fd=114))                  
UNCONN         0               0                                            91.121.165.12:53                          0.0.0.0:*             users:(("named",pid=909,fd=113))                  
UNCONN         0               0                                            91.121.165.12:53                          0.0.0.0:*             users:(("named",pid=909,fd=112))                  
UNCONN         0               0                                            91.121.165.12:53                          0.0.0.0:*             users:(("named",pid=909,fd=111))                  
UNCONN         0               0                                            91.121.165.12:53                          0.0.0.0:*             users:(("named",pid=909,fd=109))                  
UNCONN         0               0                                                127.0.0.1:53                          0.0.0.0:*             users:(("named",pid=909,fd=68))                   
UNCONN         0               0                                                127.0.0.1:53                          0.0.0.0:*             users:(("named",pid=909,fd=69))                   
UNCONN         0               0                                                127.0.0.1:53                          0.0.0.0:*             users:(("named",pid=909,fd=70))                   
UNCONN         0               0                                                127.0.0.1:53                          0.0.0.0:*             users:(("named",pid=909,fd=72))                   
UNCONN         0               0                                                127.0.0.1:53                          0.0.0.0:*             users:(("named",pid=909,fd=71))                   
UNCONN         0               0                                                127.0.0.1:53                          0.0.0.0:*             users:(("named",pid=909,fd=66))                   
UNCONN         0               0                                                127.0.0.1:53                          0.0.0.0:*             users:(("named",pid=909,fd=67))                   
UNCONN         0               0                                                127.0.0.1:53                          0.0.0.0:*             users:(("named",pid=909,fd=65))                   
UNCONN         0               0                                            127.0.0.53%lo:53                          0.0.0.0:*             users:(("systemd-resolve",pid=868,fd=12))         
UNCONN         0               0                                                    [::1]:53                             [::]:*             users:(("named",pid=909,fd=83))                   
UNCONN         0               0                                                    [::1]:53                             [::]:*             users:(("named",pid=909,fd=84))                   
UNCONN         0               0                                                    [::1]:53                             [::]:*             users:(("named",pid=909,fd=88))                   
UNCONN         0               0                                                    [::1]:53                             [::]:*             users:(("named",pid=909,fd=89))                   
UNCONN         0               0                                                    [::1]:53                             [::]:*             users:(("named",pid=909,fd=85))                   
UNCONN         0               0                                                    [::1]:53                             [::]:*             users:(("named",pid=909,fd=90))                   
UNCONN         0               0                                                    [::1]:53                             [::]:*             users:(("named",pid=909,fd=87))                   
UNCONN         0               0                                                    [::1]:53                             [::]:*             users:(("named",pid=909,fd=86))                   
UNCONN         0               0                                 [fd42:5588:e06d:ebd0::1]:53                             [::]:*             users:(("named",pid=909,fd=159))                  
UNCONN         0               0                                 [fd42:5588:e06d:ebd0::1]:53                             [::]:*             users:(("named",pid=909,fd=160))                  
UNCONN         0               0                                 [fd42:5588:e06d:ebd0::1]:53                             [::]:*             users:(("named",pid=909,fd=158))                  
UNCONN         0               0                                 [fd42:5588:e06d:ebd0::1]:53                             [::]:*             users:(("named",pid=909,fd=161))                  
UNCONN         0               0                                 [fd42:5588:e06d:ebd0::1]:53                             [::]:*             users:(("named",pid=909,fd=162))                  
UNCONN         0               0                                 [fd42:5588:e06d:ebd0::1]:53                             [::]:*             users:(("named",pid=909,fd=163))                  
UNCONN         0               0                                 [fd42:5588:e06d:ebd0::1]:53                             [::]:*             users:(("named",pid=909,fd=164))                  
UNCONN         0               0                                 [fd42:5588:e06d:ebd0::1]:53                             [::]:*             users:(("named",pid=909,fd=165))                  
UNCONN         0               0                        [fe80::216:3eff:fe24:437f]%lxdbr0:53                             [::]:*             users:(("named",pid=909,fd=177))                  
UNCONN         0               0                        [fe80::216:3eff:fe24:437f]%lxdbr0:53                             [::]:*             users:(("named",pid=909,fd=178))                  
UNCONN         0               0                        [fe80::216:3eff:fe24:437f]%lxdbr0:53                             [::]:*             users:(("named",pid=909,fd=181))                  
UNCONN         0               0                        [fe80::216:3eff:fe24:437f]%lxdbr0:53                             [::]:*             users:(("named",pid=909,fd=180))                  
UNCONN         0               0                        [fe80::216:3eff:fe24:437f]%lxdbr0:53                             [::]:*             users:(("named",pid=909,fd=176))                  
UNCONN         0               0                        [fe80::216:3eff:fe24:437f]%lxdbr0:53                             [::]:*             users:(("named",pid=909,fd=175))                  
UNCONN         0               0                        [fe80::216:3eff:fe24:437f]%lxdbr0:53                             [::]:*             users:(("named",pid=909,fd=179))                  
UNCONN         0               0                        [fe80::216:3eff:fe24:437f]%lxdbr0:53                             [::]:*             users:(("named",pid=909,fd=182))

tomp · January 13, 2021, 2:45pm

Yes, you’re only worried about the processes listening on 10.206.203.1 and fd42:5588:e06d:ebd0::1 though as those are what LXD’s dnsmasq will listen on.

You need to reconfigure named to not listen on all IPs.

NinoBYSA · January 13, 2021, 2:56pm

Thanks!

I even had the old config in /etc/bind/named.conf.options.dpkg-old since it was overwritten during the upgrade.