Cannot resolve non-LXD domains within containers after setting up .lxd DNS resolution

LXD
1

I have followed the guide that @simos wrote on his blog here: https://blog.simos.info/how-to-use-lxd-container-hostnames-on-the-host-in-ubuntu-18-04/

It works! Resolving my containers on the host (and within containers) with systemd-resolve just fine:

root@testcontainer:~# systemd-resolve othercontainer.lxd
othercontainer.lxd: fd42:86b1:4590:a8d9:216:3eff:fe8b:a44 -- link: eth0
             10.14.216.203                     -- link: eth0

-- Information acquired via protocol DNS in 2.2ms.
-- Data is authenticated: no

Unfortunately, that seems to have broken all other external name resolution within containers:

root@testcontainer:~# systemd-resolve google.com
google.com: resolve call failed: All attempts to contact name servers or networks failed

If I change the DNS servers to a non-local one (e.g. 8.8.8.8) in /etc/resolv.conf, it works. By default, it’s currently:

nameserver 127.0.0.53
options edns0
search lxd

On restart, and sporadically this gets overwritten, so it’s not exactly a permanent solution.

Within the containers, the output of systemd-resolve --status is:

root@testcontainer:~# systemd-resolve --status
Global
       LLMNR setting: no
MulticastDNS setting: no
  DNSOverTLS setting: no
      DNSSEC setting: no
    DNSSEC supported: no
          DNSSEC NTA: 10.in-addr.arpa
                      16.172.in-addr.arpa
                      168.192.in-addr.arpa
                      17.172.in-addr.arpa
                      18.172.in-addr.arpa
                      19.172.in-addr.arpa
                      20.172.in-addr.arpa
                      21.172.in-addr.arpa
                      22.172.in-addr.arpa
                      23.172.in-addr.arpa
                      24.172.in-addr.arpa
                      25.172.in-addr.arpa
                      26.172.in-addr.arpa
                      27.172.in-addr.arpa
                      28.172.in-addr.arpa
                      29.172.in-addr.arpa
                      30.172.in-addr.arpa
                      31.172.in-addr.arpa
                      corp
                      d.f.ip6.arpa
                      home
                      internal
                      intranet
                      lan
                      local
                      private
                      test

Link 98 (eth0)
      Current Scopes: DNS
DefaultRoute setting: yes
       LLMNR setting: yes
MulticastDNS setting: no
  DNSOverTLS setting: no
      DNSSEC setting: no
    DNSSEC supported: no
  Current DNS Server: 10.14.216.1
         DNS Servers: 10.14.216.1
                      fd42:86b1:4590:a8d9::1
                      fe80::ecfe:c9ff:feb3:cfb2
          DNS Domain: lxd

It appears to be a problem for other people too, there are some people in that article in the comments describing a similar scenario: https://blog.simos.info/how-to-use-lxd-container-hostnames-on-the-host-in-ubuntu-18-04/#comment-407890

How can I fix this permanently and apply this to all containers? Let me know if there’s any more info I can provide!

2

The tutorial you followed assumes that the default Ubuntu resolution applies on the host, that is, with systemd-resolve. Is your host /etc/resolv.conf pointing to 127.0.0.53 ?

3

Yes, it is.

The host’s resolv.conf is generated through systemd-resolved, containing comments as well as the following effective lines:

nameserver 127.0.0.53
options edns0
search lxd

Thanks

4

Try this on your host:

sudo systemd-resolve --interface <your-network-physical-interface> --set-domain ~.

and possibly (not sure if it’s needed in fact)

systemd-resolve --flush-caches

what gives then ?

5

Afraid that doesn’t solve the problem.

It’s worth mentioning that resolution on the host does work for both gTLDs and .lxd domains (before and after that command):

$ systemd-resolve google.com
google.com: 172.217.21.238
            2a00:1450:4001:81f::200e

-- Information acquired via protocol DNS in 2.0ms.
-- Data is authenticated: no
6

if all else fails there is always logging. you can add something like

log-queries
log-async
log-facility=/var/log/dnsmasq-lxd.log

using the same trick as for adding dns-loop-detect
then after restarting lxd do a
tail /var/log/dnsmasq-lxd.log -F
and do a resolution test in the container.(systemd-resolve linuxcontainers.org or anything)

if you get a message about request denied at this point, what gives on the host

lxc network show lxdbr0
systemd-resolve --status
ip route
7

Thanks for your help so far!

I do not think the problem is within the host - it is some problem with the lxd resolver - all dns requests on the host work perfectly fine, it is only those within the guests that start to fail. But for the outputs of those commands, see below.

Within the log, everytime i run dig myspace.com on the guests, a surprising amount of requests are made:

Apr 26 13:22:42 dnsmasq[2141]: query[A] myspace.com from 10.14.216.219
Apr 26 13:22:42 dnsmasq[2141]: query[A] myspace.com from fd42:86b1:4590:a8d9:216:3eff:fe70:c722
Apr 26 13:22:42 dnsmasq[2141]: query[A] myspace.com from fe80::216:3eff:fe70:c722
Apr 26 13:22:42 dnsmasq[2141]: query[A] myspace.com from 10.14.216.219
Apr 26 13:22:42 dnsmasq[2141]: query[A] myspace.com from fd42:86b1:4590:a8d9:216:3eff:fe70:c722
Apr 26 13:22:42 dnsmasq[2141]: query[A] myspace.com from fe80::216:3eff:fe70:c722
Apr 26 13:22:42 dnsmasq[2141]: query[A] myspace.com from 10.14.216.219
Apr 26 13:22:42 dnsmasq[2141]: query[A] myspace.com from fd42:86b1:4590:a8d9:216:3eff:fe70:c722
Apr 26 13:22:42 dnsmasq[2141]: query[A] myspace.com from fe80::216:3eff:fe70:c722
Apr 26 13:22:42 dnsmasq[2141]: query[A] myspace.com from 10.14.216.219
Apr 26 13:22:42 dnsmasq[2141]: query[A] myspace.com from fd42:86b1:4590:a8d9:216:3eff:fe70:c722
Apr 26 13:22:42 dnsmasq[2141]: query[A] myspace.com from fe80::216:3eff:fe70:c722
Apr 26 13:22:42 dnsmasq[2141]: query[A] myspace.com from 10.14.216.219
Apr 26 13:22:42 dnsmasq[2141]: query[A] myspace.com from fd42:86b1:4590:a8d9:216:3eff:fe70:c722
Apr 26 13:22:42 dnsmasq[2141]: query[A] myspace.com from fe80::216:3eff:fe70:c722
Apr 26 13:22:42 dnsmasq[2141]: query[A] myspace.com from 10.14.216.219
Apr 26 13:22:42 dnsmasq[2141]: query[A] myspace.com from fd42:86b1:4590:a8d9:216:3eff:fe70:c722
Apr 26 13:22:42 dnsmasq[2141]: query[A] myspace.com from fe80::216:3eff:fe70:c722
Apr 26 13:22:42 dnsmasq[2141]: query[A] myspace.com from 10.14.216.219
Apr 26 13:22:42 dnsmasq[2141]: query[A] myspace.com from fd42:86b1:4590:a8d9:216:3eff:fe70:c722
Apr 26 13:22:42 dnsmasq[2141]: query[A] myspace.com from fe80::216:3eff:fe70:c722
Apr 26 13:22:42 dnsmasq[2141]: query[A] myspace.com from 10.14.216.219
Apr 26 13:22:42 dnsmasq[2141]: query[A] myspace.com from fd42:86b1:4590:a8d9:216:3eff:fe70:c722
Apr 26 13:22:42 dnsmasq[2141]: query[A] myspace.com from fe80::216:3eff:fe70:c722
Apr 26 13:22:47 dnsmasq[2141]: query[A] myspace.com from 10.14.216.219
Apr 26 13:22:47 dnsmasq[2141]: query[A] myspace.com from fd42:86b1:4590:a8d9:216:3eff:fe70:c722
Apr 26 13:22:47 dnsmasq[2141]: query[A] myspace.com from fe80::216:3eff:fe70:c722
Apr 26 13:22:47 dnsmasq[2141]: query[A] myspace.com from 10.14.216.219
Apr 26 13:22:47 dnsmasq[2141]: query[A] myspace.com from fd42:86b1:4590:a8d9:216:3eff:fe70:c722
Apr 26 13:22:47 dnsmasq[2141]: query[A] myspace.com from fe80::216:3eff:fe70:c722
Apr 26 13:22:47 dnsmasq[2141]: query[A] myspace.com from 10.14.216.219
Apr 26 13:22:47 dnsmasq[2141]: query[A] myspace.com from fd42:86b1:4590:a8d9:216:3eff:fe70:c722
Apr 26 13:22:47 dnsmasq[2141]: query[A] myspace.com from fe80::216:3eff:fe70:c722
Apr 26 13:22:47 dnsmasq[2141]: query[A] myspace.com from 10.14.216.219
Apr 26 13:22:47 dnsmasq[2141]: query[A] myspace.com from fd42:86b1:4590:a8d9:216:3eff:fe70:c722
Apr 26 13:22:47 dnsmasq[2141]: query[A] myspace.com from fe80::216:3eff:fe70:c722
Apr 26 13:22:47 dnsmasq[2141]: query[A] myspace.com from 10.14.216.219
Apr 26 13:22:47 dnsmasq[2141]: query[A] myspace.com from fd42:86b1:4590:a8d9:216:3eff:fe70:c722
Apr 26 13:22:47 dnsmasq[2141]: query[A] myspace.com from fe80::216:3eff:fe70:c722
Apr 26 13:22:47 dnsmasq[2141]: query[A] myspace.com from 10.14.216.219
Apr 26 13:22:47 dnsmasq[2141]: query[A] myspace.com from fd42:86b1:4590:a8d9:216:3eff:fe70:c722
Apr 26 13:22:47 dnsmasq[2141]: query[A] myspace.com from fe80::216:3eff:fe70:c722
Apr 26 13:22:47 dnsmasq[2141]: query[A] myspace.com from 10.14.216.219
Apr 26 13:22:47 dnsmasq[2141]: query[A] myspace.com from fd42:86b1:4590:a8d9:216:3eff:fe70:c722
Apr 26 13:22:47 dnsmasq[2141]: query[A] myspace.com from fe80::216:3eff:fe70:c722
Apr 26 13:22:47 dnsmasq[2141]: query[A] myspace.com from 10.14.216.219
Apr 26 13:22:47 dnsmasq[2141]: query[A] myspace.com from fd42:86b1:4590:a8d9:216:3eff:fe70:c722
Apr 26 13:22:47 dnsmasq[2141]: query[A] myspace.com from fe80::216:3eff:fe70:c722
Apr 26 13:22:52 dnsmasq[2141]: query[A] myspace.com from 10.14.216.219
Apr 26 13:22:52 dnsmasq[2141]: query[A] myspace.com from fd42:86b1:4590:a8d9:216:3eff:fe70:c722
Apr 26 13:22:52 dnsmasq[2141]: query[A] myspace.com from fe80::216:3eff:fe70:c722
Apr 26 13:22:52 dnsmasq[2141]: query[A] myspace.com from 10.14.216.219
Apr 26 13:22:52 dnsmasq[2141]: query[A] myspace.com from fd42:86b1:4590:a8d9:216:3eff:fe70:c722
Apr 26 13:22:52 dnsmasq[2141]: query[A] myspace.com from fe80::216:3eff:fe70:c722
Apr 26 13:22:52 dnsmasq[2141]: query[A] myspace.com from 10.14.216.219
Apr 26 13:22:52 dnsmasq[2141]: query[A] myspace.com from fd42:86b1:4590:a8d9:216:3eff:fe70:c722
Apr 26 13:22:52 dnsmasq[2141]: query[A] myspace.com from fe80::216:3eff:fe70:c722
Apr 26 13:22:52 dnsmasq[2141]: query[A] myspace.com from 10.14.216.219
Apr 26 13:22:52 dnsmasq[2141]: query[A] myspace.com from fd42:86b1:4590:a8d9:216:3eff:fe70:c722
Apr 26 13:22:52 dnsmasq[2141]: query[A] myspace.com from fe80::216:3eff:fe70:c722
Apr 26 13:22:52 dnsmasq[2141]: query[A] myspace.com from 10.14.216.219
Apr 26 13:22:52 dnsmasq[2141]: query[A] myspace.com from fd42:86b1:4590:a8d9:216:3eff:fe70:c722
Apr 26 13:22:52 dnsmasq[2141]: query[A] myspace.com from fe80::216:3eff:fe70:c722
Apr 26 13:22:52 dnsmasq[2141]: query[A] myspace.com from 10.14.216.219
Apr 26 13:22:52 dnsmasq[2141]: query[A] myspace.com from fd42:86b1:4590:a8d9:216:3eff:fe70:c722
Apr 26 13:22:52 dnsmasq[2141]: query[A] myspace.com from fe80::216:3eff:fe70:c722
Apr 26 13:22:52 dnsmasq[2141]: query[A] myspace.com from 10.14.216.219
Apr 26 13:22:52 dnsmasq[2141]: query[A] myspace.com from fd42:86b1:4590:a8d9:216:3eff:fe70:c722
Apr 26 13:22:52 dnsmasq[2141]: query[A] myspace.com from fe80::216:3eff:fe70:c722
Apr 26 13:22:52 dnsmasq[2141]: query[A] myspace.com from 10.14.216.219
Apr 26 13:22:52 dnsmasq[2141]: query[A] myspace.com from fd42:86b1:4590:a8d9:216:3eff:fe70:c722
Apr 26 13:22:52 dnsmasq[2141]: query[A] myspace.com from fe80::216:3eff:fe70:c722

This was for a single run.

Output of the various commands:

lxc network show lxdbr0
    config:
      ipv4.address: 10.14.216.1/24
      ipv4.nat: "true"
      ipv6.address: fd42:86b1:4590:a8d9::1/64
      ipv6.nat: "true"
      raw.dnsmasq: |
        auth-zone=lxd
        dns-loop-detect
        log-queries
        log-async
        log-facility=/var/log/dnsmasq-lxd.log
    description: ""
    name: lxdbr0
    type: bridge
    used_by:
    - /1.0/containers/container1
    - /1.0/containers/container2
    - /1.0/containers/container3
    - /1.0/containers/container4
    - /1.0/containers/container5
    - /1.0/containers/container6
    - /1.0/containers/container7
    - /1.0/containers/container8
    managed: true
    status: Created
    locations:
    - none

systemd-resolve --status (there’s quite a few irrelevant entries, of the form veth*):

Global
         DNS Servers: 213.133.99.99
                      213.133.100.100
                      213.133.98.98
          DNSSEC NTA: 10.in-addr.arpa
                      16.172.in-addr.arpa
                      168.192.in-addr.arpa
                      17.172.in-addr.arpa
                      18.172.in-addr.arpa
                      19.172.in-addr.arpa
                      20.172.in-addr.arpa
                      21.172.in-addr.arpa
                      22.172.in-addr.arpa
                      23.172.in-addr.arpa
                      24.172.in-addr.arpa
                      25.172.in-addr.arpa
                      26.172.in-addr.arpa
                      27.172.in-addr.arpa
                      28.172.in-addr.arpa
                      29.172.in-addr.arpa
                      30.172.in-addr.arpa
                      31.172.in-addr.arpa
                      corp
                      d.f.ip6.arpa
                      home
                      internal
                      intranet
                      lan
                      local
                      private
                      test

Link 11 (lxdbr0)
      Current Scopes: DNS
       LLMNR setting: yes
MulticastDNS setting: no
      DNSSEC setting: no
    DNSSEC supported: no
         DNS Servers: 10.14.216.1
          DNS Domain: lxd

Link 6 (lxcbr0)
      Current Scopes: none
       LLMNR setting: yes
MulticastDNS setting: no
      DNSSEC setting: no
    DNSSEC supported: no

Link 2 (eth0)
      Current Scopes: none
       LLMNR setting: yes
MulticastDNS setting: no
      DNSSEC setting: no
    DNSSEC supported: no
          DNS Domain: ~.

ip route:

default via 172.31.1.1 dev eth0
10.0.3.0/24 dev lxcbr0 proto kernel scope link src 10.0.3.1 
10.14.216.0/24 dev lxdbr0 proto kernel scope link src 10.14.216.1 
172.31.1.1 dev eth0 scope link
8

I don’t understand why it does not work for you; it’s better IMO to do it my way with DNS scope, servers and default ‘DNS route’ set on the interface instead of globally since it avoid unnecessary queries that are doomed to fail, but it should work.
Did you remove the default symlink /etc/resolv.conf and replaced it with a regular file? See http://manpages.ubuntu.com/manpages/xenial/man8/systemd-resolved.service.8.html on why it’s not a good idea.

9

Nope, I didn’t replace it, it’s still the default.

I ended up sort of temporarily solving it by just reverting some of the changes, removing the startup scripts installed via that tutorial, and running on the host:

/usr/bin/systemd-resolve --interface lxdbr0 --revert

I’m not sure this counts as a solution though, it just reverts the ability for the host to do .lxd resolution (there are changes cached, which still work, but spawning a new container and performing name resolution on it fails). Using the lxd nameserver still works (i.e. running dig @10.14.216.1 containername.lxd), but it’s not really usable because it means I can’t resolve .lxd on the host using the system resolver.

Any tips on how to configure the host to allow .lxd resolution?

10

I was asking because I see a ‘search lxd’ that is not in my default resolv.conf on the only 18.04 system and default systemd where I can dabble with network resolution and where I tested @simos tutorial. I don’t know how it could be a following of the tutorial. Anyway, I’d say that you should attempt to set your dns servers on your physical interface, not globally. It’s the more satisfying setup and the only one where you can be sure to avoid loops.
Something like that;
Link 2 (enp3s0)
Current Scopes: DNS
(…)
DNS Servers: 10.0.0.250
DNS Domain: ~.

here I use a private DNS server on my internal network but you could do the same with public DNS servers.
Other than that you can make do with solutions on this thread
These are not very pretty hacks but you may care only about the result. Try to not use 8.8.8.8 at least.