Empty resolv.conf in containers after host reboot

LXD
5

I’ll try to provide additional details just in case others come across this post: this problem is a little bit similar to this one, although the cause may be different.

On the host, systemd-resolved shows this:

$ sudo systemctl status systemd-resolved
● systemd-resolved.service - Network Name Resolution
     Loaded: loaded (/lib/systemd/system/systemd-resolved.service; enabled; vendor preset: enabled)
     Active: active (running) since Wed 2022-08-17 16:15:09 UTC; 45min ago
       Docs: man:systemd-resolved.service(8)
             man:org.freedesktop.resolve1(5)
             https://www.freedesktop.org/wiki/Software/systemd/writing-network-configuration-managers
             https://www.freedesktop.org/wiki/Software/systemd/writing-resolver-clients
   Main PID: 1200 (systemd-resolve)
     Status: "Processing requests..."
      Tasks: 1 (limit: 38123)
     Memory: 5.8M
        CPU: 708ms
     CGroup: /system.slice/systemd-resolved.service
             └─1200 /lib/systemd/systemd-resolved

Aug 17 16:15:09 jpbm systemd[1]: Starting Network Name Resolution...
Aug 17 16:15:09 jpbm systemd-resolved[1200]: Positive Trust Anchors:
Aug 17 16:15:09 jpbm systemd-resolved[1200]: . IN DS 20326 8 2 e06d44b80b8f1d39a95c0b0d7c65d08458e880409bbc683457104237c7f8ec8d
Aug 17 16:15:09 jpbm systemd-resolved[1200]: Negative trust anchors: home.arpa 10.in-addr.arpa 16.172.in-addr.arpa 17.172.in-addr.arpa 18.172.in-addr.arpa 19.172.in-addr.arpa 20.172.in-addr.arpa 21.172.in-addr.arpa 22.172.in-addr.arpa 23.172.in-addr.arpa 24.172.in-addr.arpa 25.172.in-addr.arpa 26.>
Aug 17 16:15:09 jpbm systemd-resolved[1200]: Using system hostname 'jpbm'.
Aug 17 16:15:09 jpbm systemd[1]: Started Network Name Resolution.
Aug 17 16:26:46 jpbm systemd-resolved[1200]: lxdbr0: Bus client set DNS server list to: 1.1.1.1
Aug 17 16:52:37 jpbm systemd-resolved[1200]: Using degraded feature set UDP instead of UDP+EDNS0 for DNS server 1.1.1.1.

There’s also dnsmasq running:

$ ps -ef | grep masq
lxd         2243    2138  0 16:15 ?        00:00:00 dnsmasq --keep-in-foreground --strict-order --bind-interfaces --except-interface=lo --pid-file= --no-ping --interface=lxdbr0 --dhcp-rapid-commit --quiet-dhcp --quiet-dhcp6 --quiet-ra --listen-address=192.168.1.1 --dhcp-no-override --dhcp-authoritative --dhcp-leasefile=/var/snap/lxd/common/lxd/networks/lxdbr0/dnsmasq.leases --dhcp-hostsfile=/var/snap/lxd/common/lxd/networks/lxdbr0/dnsmasq.hosts --dhcp-option-force=26,1450 --dhcp-range 192.168.1.2,192.168.1.5,1h -s lxd --interface-name _gateway.lxd,lxdbr0 -S /lxd/ --conf-file=/var/snap/lxd/common/lxd/networks/lxdbr0/dnsmasq.raw -u lxd -g lxd

On the host DNS resolution works fine and resolv.conf points to stub-resolv.conf:

$ sudo dir -lat /etc/resolv.conf
lrwxrwxrwx 1 root root 39 Feb 28 11:00 /etc/resolv.conf -> ../run/systemd/resolve/stub-resolv.conf

$ resolvectl status
Global
       Protocols: -LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported
resolv.conf mode: stub

Link 2 (enp1s0f0)
    Current Scopes: DNS
         Protocols: +DefaultRoute +LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported
Current DNS Server: 1.0.0.1
       DNS Servers: 1.1.1.1 1.0.0.1

Link 3 (enp1s0f1)
Current Scopes: none
     Protocols: -DefaultRoute +LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported

Link 4 (lxdbr0)
    Current Scopes: DNS
         Protocols: +DefaultRoute +LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported
Current DNS Server: 1.1.1.1
       DNS Servers: 1.1.1.1

Inside of a VM (also Ubuntu 22.04) bridged to lxdbr0, ../run/systemd/resolve/stub-resolv.conf is empty and /etc/resolv.conf symlinking to it is also empty.

I tried that today:

$ lxc launch ubuntu:22.04 dnstest
Creating dnstest
Starting dnstest                              

ubuntu@bm:~$ lxc shell dnstest
root@dnstest:~# nslookup www.google.com
;; communications error to 127.0.0.1#53: connection refused

root@dnstest:~# dir -lat /etc/resolv.conf 
lrwxrwxrwx 1 root root 39 Aug 10 06:56 /etc/resolv.conf -> ../run/systemd/resolve/stub-resolv.conf
root@dnstest:~# sudo cat ../run/systemd/resolve/stub-resolv.conf
sudo: unable to resolve host dnstest: Temporary failure in name resolution
cat: ../run/systemd/resolve/stub-resolv.conf: No such file or directory

root@dnstest:~# sudo systemctl status systemd-resolved
sudo: unable to resolve host dnstest: Temporary failure in name resolution
○ systemd-resolved.service - Network Name Resolution
     Loaded: loaded (/lib/systemd/system/systemd-resolved.service; enabled; vendor preset: enabled)
     Active: inactive (dead)
       Docs: man:systemd-resolved.service(8)
             man:org.freedesktop.resolve1(5)
             https://www.freedesktop.org/wiki/Software/systemd/writing-network-configuration-managers
             https://www.freedesktop.org/wiki/Software/systemd/writing-resolver-clients

root@dnstest:~# sudo systemctl start systemd-resolved
sudo: unable to resolve host dnstest: Temporary failure in name resolution

root@dnstest:~# sudo systemctl status systemd-resolved
● systemd-resolved.service - Network Name Resolution
     Loaded: loaded (/lib/systemd/system/systemd-resolved.service; enabled; vendor preset: enabled)
     Active: active (running) since Wed 2022-08-17 17:07:32 UTC; 2s ago
       Docs: man:systemd-resolved.service(8)
             man:org.freedesktop.resolve1(5)
             https://www.freedesktop.org/wiki/Software/systemd/writing-network-configuration-managers
             https://www.freedesktop.org/wiki/Software/systemd/writing-resolver-clients
   Main PID: 320 (systemd-resolve)
     Status: "Processing requests..."
      Tasks: 1 (limit: 38123)
     Memory: 4.4M
        CPU: 27ms
     CGroup: /system.slice/systemd-resolved.service
             └─320 /lib/systemd/systemd-resolved

Aug 17 17:07:32 dnstest systemd[1]: Starting Network Name Resolution...
Aug 17 17:07:32 dnstest systemd-resolved[320]: Positive Trust Anchors:
Aug 17 17:07:32 dnstest systemd-resolved[320]: . IN DS 20326 8 2 e06d44b80b8f1d39a95c0b0d7c65d08458e880409bbc683457104237c7f8ec8d
Aug 17 17:07:32 dnstest systemd-resolved[320]: Negative trust anchors: home.arpa 10.in-addr.arpa 16.172.in-addr.arpa 17.172.in-addr.arpa 18.172.in-addr.arpa 19.172.in-addr.arpa 20.172.in-addr.arpa 21.172.in-addr.arpa 22.172.in-addr.arpa 23.172.in-addr.arpa 24.172.in-addr.arpa 25.172.in-addr.arpa 2>
Aug 17 17:07:32 dnstest systemd-resolved[320]: Using system hostname 'dnstest'.
Aug 17 17:07:32 dnstest systemd[1]: Started Network Name Resolution.

If I now start systemd-resolved and cat /etc/resolv.conf (i.e. the file it’s symlinked to), now it’s not empty.

root@dnstest:~# dir -lat /etc/resolv.conf 
lrwxrwxrwx 1 root root 39 Aug 10 06:56 /etc/resolv.conf -> ../run/systemd/resolve/stub-resolv.conf

root@dnstest:~# cat ../run/systemd/resolve/stub-resolv.conf

nameserver 127.0.0.53
options edns0 trust-ad
search .

root@dnstest:~# nslookup www.google.com
Server:		127.0.0.53
Address:	127.0.0.53#53

** server can't find www.google.com: SERVFAIL

root@dnstest:~# sudo systemctl status systemd-resolved
● systemd-resolved.service - Network Name Resolution
     Loaded: loaded (/lib/systemd/system/systemd-resolved.service; enabled; vendor preset: enabled)
     Active: active (running) since Wed 2022-08-17 17:07:32 UTC; 39s ago
       Docs: man:systemd-resolved.service(8)
             man:org.freedesktop.resolve1(5)
             https://www.freedesktop.org/wiki/Software/systemd/writing-network-configuration-managers
             https://www.freedesktop.org/wiki/Software/systemd/writing-resolver-clients
   Main PID: 320 (systemd-resolve)
     Status: "Processing requests..."
      Tasks: 1 (limit: 38123)
     Memory: 4.4M
        CPU: 27ms
     CGroup: /system.slice/systemd-resolved.service
             └─320 /lib/systemd/systemd-resolved

Aug 17 17:07:32 dnstest systemd[1]: Starting Network Name Resolution...
Aug 17 17:07:32 dnstest systemd-resolved[320]: Positive Trust Anchors:
Aug 17 17:07:32 dnstest systemd-resolved[320]: . IN DS 20326 8 2 e06d44b80b8f1d39a95c0b0d7c65d08458e880409bbc683457104237c7f8ec8d
Aug 17 17:07:32 dnstest systemd-resolved[320]: Negative trust anchors: home.arpa 10.in-addr.arpa 16.172.in-addr.arpa 17.172.in-addr.arpa 18.172.in-addr.arpa 19.172.in-addr.arpa 20.172.in-addr.arpa 21.172.in-addr.arpa 22.172.in-addr.arpa 23.172.in-addr.arpa 24.172.in-addr.arpa 25.172.in-addr.arpa 2>
Aug 17 17:07:32 dnstest systemd-resolved[320]: Using system hostname 'dnstest'.
Aug 17 17:07:32 dnstest systemd[1]: Started Network Name Resolution.

If I shut down this VM, and start it again, after login DNS resolution still doesn’t work and systemd-resolved is dead. If I start systemd-resolved manually, DNS still fails the same as before, so it’s consistent:

  • systemd-resolved is dead upon startup even if it was enabled
  • if it’s dead, ../run/systemd/resolve/stub-resolv.conf is empty, if it’s running, ../run/systemd/resolve/stub-resolv.conf is not empty - same as when I enabled it in the VM earlier)
  • either way, DNS resolution does not work regardless of whether systemd-resolved is running or not in guest VM. Even when I removed /etc/resolv.conf and created a new file rather than symlink, with nameserver 1.1.1.1 in it, resolution in guest VM still didn’t work (which I find strange - I have that “workaround” work in other VMs on the same host, but I created those several weeks ago, maybe they are slightly different in some way)

Edit: I should also post this, as I’ve re-installed OS since that and the problem persists with another version (host OS is still Jammy). LXD is still running from snapd, but this version:

$ snap list --all lxd
Name  Version        Rev    Tracking    Publisher   Notes
lxd   5.0.0-b0287c1  22923  5.0/stable  canonical✓  -