Ubuntu 18.04 infinite Loop dnsmasq systemd-resolved

rudiservo · September 25, 2018, 5:03pm

Hi
I followed Simos blog on how to add hostname resolve from your host to the container.

It works fine but one issue, it goes on infinite loop between dnsmasq and resolved and uses quite a lot of the cpu.

Anyone has a fix for this?

simos · September 25, 2018, 8:20pm

Hi!

You are the second person reporting such a result. It happened to work for me when I tested it but did not keep it running to notice if there was a CPU-load issue.

What’s needed, is to figure out why this issue is really happening so that to find what to do to fix it.

rudiservo · September 25, 2018, 8:47pm

Ok lets do it!

It was a fresh instalation of Xubuntu 18.04 64bit with Luks and ext4.

installed with apt
lxd, strongsan

did had docker but purged it just in case, tried debuging installing dnsmasq daemon, purge it after.

These are my config files

/etc/dnsmasq.d/lxd
server=/lxd/10.146.38.1
bind-interfaces
except-interface=lxdbr0

lxd network
config:
ipv4.address: 10.146.38.1/24
ipv4.nat: “true”
ipv6.address: fd42:82c5:af81:fa56::1/64
ipv6.nat: “true”
description: “”
name: lxdbr0
type: bridge
used_by:

/1.0/containers/d0
/1.0/containers/d1
/1.0/containers/d2
managed: true
status: Created
locations:
none

rudiservo · September 26, 2018, 4:33pm

just tried in a fresh system with Xubuntu 18.04 and same thing, just installed LXD package nothing else.
Something to do with network manager maybe?

Already did try out you approach on DNS for LXC containers
but to it didn’t work.
My limitation is not knowing a thing about systemd-resolved or network-manager and how they interact, my only other choice that I am not very fan of is disabling systemd-resolved and installing dnsmasq for the hole system.

dnsmasq has got a loop-detection option but I do not know how to activate it on the LXD side of things.

My understanding of the problem would be that resolved asks dnsmasq, and since dnsmasq also resolves names for the containers it might ask again to resolved and get on a loop, I will wireshark the lxdbr0 port to confirm my suspition.
This does not happens only when do dig d0.lxd, it goes on a loop just by dig google.com or any not cached dns record.

I will search for an option to only query a dns for a specific TLD.

rudiservo · September 26, 2018, 5:02pm

Found something, the loop is an IPv6 query that if it can’t find (because it does not exists) and goes on a infinite loop between 127.0.0.1, 127.0.0.53 and lxdbr0 ip.
Also there are and extra set of initial queries between these 3 of IPv4 with correct answers, so a dns query is made to everyone regardless.

simos · September 26, 2018, 5:37pm

Nice!

I just tried with IPv6 working, and I did not see any extra queries.

So the issue is to figure what types of IPv6 misconfiguration would make this problem appear.

rudiservo · September 26, 2018, 7:46pm

So disable ipv6 on the network-manager?

rudiservo · September 26, 2018, 11:40pm

ok so I found these two bug reports on ubuntu.

https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1694156
https://bugs.launchpad.net/ubuntu/+source/dnsmasq/+bug/1672099

one solution provided was to use

no-resolv
bind-interfaces
interface=lo
server=127.0.0.53

Issue is I don’t have dnsmasq installed on my machine, I have dnsmasq-base that comes has a lxd dependency.
I trie every configuration possible in /etc/dnsmasq.d/lxd and it doesnt seem to grab any of it

so how do I solve this? do I try and do some stuff in /var/lib/lxd/networks/lxdbr0/dnsmasq.raw
is this the “correct” config file to mess around or do I install dnsmasq package and mess around in /etc/dnsmasq.d/lxd

dnsmasq --strict-order --bind-interfaces --pid-file=/var/lib/lxd/networks/lxdbr0/dnsmasq.pid --except-interface=lo --interface=lxdbr0 --quiet-dhcp --quiet-dhcp6 --quiet-ra --listen-address=10.146.38.1 --dhcp-no-override --dhcp-authoritative --dhcp-leasefile=/var/lib/lxd/networks/lxdbr0/dnsmasq.leases --dhcp-hostsfile=/var/lib/lxd/networks/lxdbr0/dnsmasq.hosts --dhcp-range 10.146.38.2,10.146.38.254,1h -s lxd -S /lxd/ --conf-file=/var/lib/lxd/networks/lxdbr0/dnsmasq.raw -u lxd

sorry for being a pain, just been around this for the last couple of days.

simos · September 27, 2018, 6:52pm

I had a look at those two bug reports on Launchpad. The first references an additional report,

which describes a similar issue that we have with LXD but they get this with libvirt.
Here’s the text,

In debugging bug #1694156, I found that ultimately my problem was triggered by a hard-coded /etc/resolvconf/resolv.conf.d/tail I had set once upon a time pointing to my libvirt dnsmasq server. It should not be necessary to manually edit /etc/resolvconf/resolv.conf.d/tail to register dnsmasq; instead, on a system where systemd-resolved is running, libvirt should use the DBUS protocol to register its dnsmasq with systemd-resolved, specifying both SetLinkDNS and SetLinkDomains. This would enable properly-scoped DNS lookups for only the hosts on the libvirt bridge, avoiding any possibility of DNS loops and avoiding the need for manual configuration.

To do this properly, libvirt does need to declare a link domain (SetLinkDomains) that doesn’t conflict with other public DNS, or other non-authoritative DNS that may be configured on the system. I would suggest using just ‘libvirt.’ as a TLD, by default.

For example implementation, please see ./src/dns-manager/nm-dns-systemd-resolved.c:send_updates() in the network-manager source.

It describes how to do this programmatically, which means that there should be a way to interpret as additional command-line options to the configuration we do for LXD.

rudiservo · September 27, 2018, 7:25pm

a quick fix for now would be to add to the this “–dns-loop-detect” to the dnsmasq at startup.

As for the rest, I do not know where to start fixing in LXD, has much has I would like to, it would probably take me more then a few weeks full time.

Should I open a ticket in github?

simos · September 27, 2018, 9:28pm

Here is a similar report regarding K8s,

github.com/kubernetes/kubernetes

Please add --dns-loop-detect option to dnsmasq run by kube-dns

opened 01:14PM - 11 Aug 18 UTC

closed 03:47PM - 18 Aug 18 UTC

hedayat

kind/bug sig/network

**Is this a BUG REPORT or FEATURE REQUEST?**: > Uncomment only one, leave it …on its own line: > /kind bug > /kind feature **What happened**: If hosts `/etc/resolv.conf` points to a local DNS server, or if it contains the cluster's DNS server (to enable looking up services from host), it'll cause DNS loop which can effectively disable kube-dns or make it very slow. **What you expected to happen**: At least, kube-dns should detect localhost or cluster DNS server entries in resolv.conf and not send queries to them. **How to reproduce it (as minimally and precisely as possible)**: Use a system with local cashing DNS server, or add cluster's DNS server address to hosts resolv.conf. **Anything else we need to know?**: According to `dnsmasq` documentation, adding `--dns-loop-detect` should make dnsmasq detect that and disable the upstream DNS server which cause the loop. It should prevent DNS from becoming unresponsive in the first case (a better solution is to extract real upstream servers, which probably cannot be done for every configuration); and for the second case it should skip sending queries to itself and send to real upstream servers.

It explains the issue and it’s relevant to LXD.

Below is the man-page of dnsmasq that explains what --dns-loop-detect does.

–dns-loop-detect

Enable code to detect DNS forwarding loops; ie the situation where a query sent to one of the upstream server eventually returns as a new query to the dnsmasq instance. The process works by generating TXT queries of the form <hex>.test and sending them to each upstream server. The hex is a UID which encodes the instance of dnsmasq sending the query and the upstream server to which it was sent. If the query returns to the server which sent it, then the upstream server through which it was sent is disabled and this event is logged. Each time the set of upstream servers changes, the test is re-run on all of them, including ones which were previously disabled.
Source: Man page of DNSMASQ

I do not think that adding ‘–dns-loop-detect’ would have a detrimental effect to the dnsmasq in LXD.

(The rest is about being able to access programmatically systemd-resolved over DBUS and set the configuration there to fix this.)

Therefore, do open a ticket on github.

As title, you may use something like: Please add --dns-loop-detect option to dnsmasq run by LXD
Mention that you have tried this and it worked for you. Also, point to this discussion.

rudiservo · September 27, 2018, 9:35pm

Thanks for the help, will do

mdfrg · October 11, 2018, 7:05pm

The solution posted by Stuart Langridge on stackexchange worked for me:

lxc network edit lxdbr0:

config:
  ipv4.address: 10.216.134.1/24
  ipv4.nat: "true"
  ipv6.address: none
  ipv6.nat: "true"
  raw.dnsmasq: |
    auth-zone=lxd
    dns-loop-detect
name: lxdbr0
type: bridge

Add the 3 lines starting with raw.dnsmasq.

@simos,

the /lib/systemd/system/lxd-host-dns.service syntax you put in your blogpost didn’t work for me under Ubuntu 18.04. Here is what worked:

[Unit]
Description=LXD host DNS service
After=multi-user.target

[Service]
Type=simple
ExecStart=/usr/local/bin/lxdhostdns_start.sh
RemainAfterExit=true
ExecStop=/usr/local/bin/lxdhostdns_stop.sh
StandardOutput=journal

[Install]
WantedBy=multi-user.target

Notice the Type=simple and After= change.

Thank you for your blog. There are little information about LXD and what you provide is very informative.

simos · October 12, 2018, 9:36am

Thanks for this.

I updated the post at https://blog.simos.info/how-to-use-lxd-container-hostnames-on-the-host-in-ubuntu-18-04/
to include

the additions of auth-zone=lxd and detect-dns-loop in the LXD managed network interface lxdbr0.
the changes to the systemd service file.

Please report if I missed anything from the blog post or I need to explain something better.

rudiservo · October 16, 2018, 2:52pm

you can do this and it will work out of the box without any services for resolved.

mkdir /etc/systemd/resolved.conf.d
nano /etc/systemd/resolved.conf.d/lxdbr0.conf

and paste this inside

[Resolve]
DNS=10.146.38.1 #the ip of the lxbr0
Domains=lxd

also what i found that work was to add nameserver to raw.dnsmasq server=8.8.8.8 because with the loop-dettect will also block normal query to any other dns, google, ubuntu, centos, etc, havent tried the auth-zone=lxd yet.

karjala · October 20, 2018, 6:13pm

I tried the solutions in the immediately-previous 2 comments in this page.

@simos, the (permanent, updated) solution that appears in your blogpost failed (on my fresh Ubuntu MATE 18.04 with snap LXD 3.6).

@rudiservo, your 3-line solution seems to work on my (completely fresh, again) system.

@simos, the failure of the “permanent solution” on the blogpost can be seen here (I executed this command right after a boot):

root@ubuntu:~# systemctl status lxd-host-dns.service 
● lxd-host-dns.service - LXD host DNS service
   Loaded: loaded (/etc/systemd/system/lxd-host-dns.service; enabled; vendor preset: enabled)
   Active: failed (Result: exit-code) since Sat 2018-10-20 10:46:39 PDT; 4min 39s ago
  Process: 1745 ExecStart=/usr/local/bin/lxdhostdns_start.sh (code=exited, status=1/FAILURE)
 Main PID: 1745 (code=exited, status=1/FAILURE)

Oct 20 10:46:39 ubuntu systemd[1]: Started LXD host DNS service.
Oct 20 10:46:39 ubuntu lxdhostdns_start.sh[1745]: Device "lxdbr0" does not exist.
Oct 20 10:46:39 ubuntu lxdhostdns_start.sh[1745]: Unknown interface lxdbr0: No such device
Oct 20 10:46:39 ubuntu systemd[1]: lxd-host-dns.service: Main process exited, code=exited, status=1/FAILURE
Oct 20 10:46:39 ubuntu systemd[1]: lxd-host-dns.service: Failed with result 'exit-code'.
root@ubuntu:~#

@simos, would it be possible please to examine @rudiservo’s solution, if it’s good?

simos · October 20, 2018, 6:47pm

It says that it cannot find a lxdbr0 network interface. Perhaps it is lxdbr1 in your case or something else?

karjala · October 20, 2018, 7:49pm

No, that is not the case. See here for proof (operating on the VM that was based on your blogpost). Also, your temporary solution works, just not your permanent one:

user@ubuntu:~$ systemctl status lxd-host-dns.service 
● lxd-host-dns.service - LXD host DNS service
   Loaded: loaded (/etc/systemd/system/lxd-host-dns.service; enabled; vendor preset: enabled)
   Active: failed (Result: exit-code) since Sat 2018-10-20 12:46:19 PDT; 1min 5s ago
  Process: 1645 ExecStart=/usr/local/bin/lxdhostdns_start.sh (code=exited, status=1/FAILURE)
 Main PID: 1645 (code=exited, status=1/FAILURE)

Oct 20 12:46:19 ubuntu systemd[1]: Started LXD host DNS service.
Oct 20 12:46:19 ubuntu lxdhostdns_start.sh[1645]: Device "lxdbr0" does not exist.
Oct 20 12:46:19 ubuntu lxdhostdns_start.sh[1645]: Unknown interface lxdbr0: No such device
Oct 20 12:46:19 ubuntu systemd[1]: lxd-host-dns.service: Main process exited, code=exited, status=1/FAILURE
Oct 20 12:46:19 ubuntu systemd[1]: lxd-host-dns.service: Failed with result 'exit-code'.
user@ubuntu:~$ lxc network list
+--------+----------+---------+-------------+---------+
|  NAME  |   TYPE   | MANAGED | DESCRIPTION | USED BY |
+--------+----------+---------+-------------+---------+
| ens33  | physical | NO      |             | 0       |
+--------+----------+---------+-------------+---------+
| lxdbr0 | bridge   | YES     |             | 1       |
+--------+----------+---------+-------------+---------+
user@ubuntu:~$

mdfrg · November 21, 2018, 11:40pm

Actually, even this hogs CPU at 100% after a while. Start pinging .lxd domains, wait 2-3min and start to see the CPU spike.

karjala · January 5, 2019, 10:17am

After having used simos’s solution in his blogpost, together with the detect-dns-loop option, I cannot do nslookup any non-lxd domain from inside an lxd container. I use ubuntu 18.04 server. Is that your experience too? Can you nslookup non-lxd domains from inside an lxd container?

Thanks