The initial wait, during the first DNS request of every container’s name, makes me want to replace systemd-resolved with dnsmasq on the host. I wonder if it’s a good idea.
You have a typo in your tutorial: You say:
(there’s an extra i)
Thanks for reporting the typo. I just fixed it.
Indeed, there is an initial wait for the timeout of the default DNS server.
I updated the blog post with a troubleshooting section, which describes the issue.
There should be a way for systemd-resolved to just ask the
lxdbr0 DNS server and not need to also consult the default DNS server. systemd-resolved can deduce that because we specified that the .lxd domains are served by the
lxdbr0 DNS server.
There should either be some special configuration in systemd for this, or there is a need for a bug report on systemd.
@simos, I tried your suggestion to do: systemctl | grep lxd, in order to find a suitable unit to place in the After= field.
I tried placing all of them in After=, did systemctl daemon-reload after each one, but your service always failed during boot time. The error message is this:
Ιουν 25 09:45:50 karjala-vmware lxdhostdns_start.sh: Device "lxdbr0" does not exist. Ιουν 25 09:45:50 karjala-vmware lxdhostdns_start.sh: Unknown interface lxdbr0: No such device
So we still haven’t solved the problem for Snap users (which is the most important), only for Apt users.
I wonder if it’s a good solution to replace systemd-resolved with dnsmasq on the host, like I had in Ubuntu 16.04. What do you think about this?
However, when I systemctl start your service after boot, and after I have logged in, that works, and the LXD containers hostnames resolve.
First of all, thank for trying out with the different options for
Ideally, it should be better to figure out a way to make it work reliably on Ubuntu 18.04 without additional system changes.
I’ll look into this a bit later on a fresh 18.04 install. If you need to get a system working at once, then go ahead and switch to
P.S. I didn’t use StartAfter= option, but the After= option. Did I do it wrong?
No, you did right. There is no StartAfter, it is
After. I just fixed my typo above.
Hey @simos, I keep having problems with this when using the fan networking and snap. I reported this initially under https://github.com/lxc/lxd/issues/4631 but after two weeks to and fro of tweaking and testing, I couldn’t move forward at all. The problem is, that lxd snap starts before network is ready, here the relevant syslog entry
Jun 27 07:20:13 kai02 lxd.daemon: err="listen tcp 192.168.88.102:8443: bind: cannot assign requested address" lvl= Jun 27 07:20:14 kai02 systemd-networkd: enp0s31f6: Gained carrier Jun 27 07:20:14 kai02 kernel: e1000e: enp0s31f6 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: None Jun 27 07:20:14 kai02 kernel: IPv6: ADDRCONF(NETDEV_CHANGE): enp0s31f6: link becomes ready Jun 27 07:20:15 kai02 systemd-networkd: enp0s31f6: Gained IPv6LL Jun 27 07:20:15 kai02 systemd-timesyncd: Network configuration changed, trying to establish connection. Jun 27 07:20:18 kai02 lxd.daemon: lvl=warn msg="Raft: Heartbeat timeout from \"\" reached, starting election" t=20 Jun 27 07:20:21 kai02 systemd-networkd: enp0s31f6: DHCPv4 address 192.168.88.102/24 via 192.168.88.1
@stgraber suggested on github to block lxd from starting until network is up, so what I did was to start with adding an ExecStartPre=/bin/sleep 20 … this helped but this doesnt really solve anything, right? So I ended up with ExecStartPre=/lib/systemd/systemd-networkd-wait-online --interface=enp0s31f6 … here the full unit file:
$ cat snap.lxd.daemon.service [Unit] # Auto-generated, DO NOT EDIT Description=Service for snap application lxd.daemon Requires=snap-lxd-7586.mount Wants=network-online.target After=snap-lxd-7586.mount network-online.target X-Snappy=yes [Service] ExecStartPre=/lib/systemd/systemd-networkd-wait-online --interface=enp0s31f6 ExecStart=/usr/bin/snap run lxd.daemon SyslogIdentifier=lxd.daemon Restart=always WorkingDirectory=/var/snap/lxd/7586 ExecStop=/usr/bin/snap run --command=stop lxd.daemon ExecReload=/usr/bin/snap run --command=reload lxd.daemon TimeoutStopSec=100 Type=simple [Install] WantedBy=multi-user.target
but this actually fails to do the trick. I also modified the unit to wait for lxdfan0 to start, because I had the same problem with the unit starting too early. I changed the after to snap.lxd.daemon.service (not sure where you get the lxd-containers.service, but that probably applies if you build from source, right?)
$ cat /lib/systemd/system/lxd-host-dns.service [Unit] Description=LXD host DNS service After=snap.lxd.daemon.service [Service] Type=oneshot ExecStartPre=/lib/systemd/systemd-networkd-wait-online --interface=lxdfan0 ExecStart=/usr/local/bin/lxdhostdns_start.sh RemainAfterExit=true ExecStop=/usr/local/bin/lxdhostdns_stop.sh StandardOutput=journal [Install] WantedBy=multi-user.target
i naturally modified the start/stop scripts to use lxdfan0. Again, this runs for me if i use a sleep here, but using systemd-networkd-wait-online fails with
-- Unit lxd-host-dns.service has begun starting up. Jun 27 07:36:28 kai02 systemd-networkd-wait-online: ignoring: veth3XRAOP Jun 27 07:36:28 kai02 systemd-networkd-wait-online: ignoring: vethPB44BR Jun 27 07:36:28 kai02 systemd-networkd-wait-online: ignoring: enp0s31f6 Jun 27 07:36:28 kai02 systemd-networkd-wait-online: ignoring: vethM1P8JU Jun 27 07:36:28 kai02 systemd-networkd-wait-online: ignoring: veth7KP1IC Jun 27 07:36:28 kai02 systemd-networkd-wait-online: managing: lxdfan0 Jun 27 07:36:28 kai02 systemd-networkd-wait-online: ignoring: lo Jun 27 07:36:28 kai02 systemd-networkd-wait-online: ignoring: lxdfan0-mtu Jun 27 07:36:28 kai02 systemd-networkd-wait-online: ignoring: veth21DABR Jun 27 07:36:28 kai02 systemd-networkd-wait-online: ignoring: lxdfan0-fan Jun 27 07:36:28 kai02 systemd-networkd-wait-online: ignoring: vethXHDGFH Jun 27 07:36:28 kai02 systemd-networkd-wait-online: ignoring: wlp1s0 Jun 27 07:36:28 kai02 systemd-networkd-wait-online: ignoring: vethECXIQM Jun 27 07:36:28 kai02 systemd-networkd-wait-online: ignoring: veth0BGHQS Jun 27 07:36:28 kai02 lxdhostdns_start.sh: The specified interface lxdfan0 is managed by systemd-networkd. Operation refused. Jun 27 07:36:28 kai02 lxdhostdns_start.sh: Please configure DNS settings for systemd-networkd managed interfaces directly in their .network files. Jun 27 07:36:28 kai02 systemd: lxd-host-dns.service: Main process exited, code=exited, status=1/FAILURE Jun 27 07:36:28 kai02 systemd: lxd-host-dns.service: Failed with result 'exit-code'. Jun 27 07:36:28 kai02 systemd: Failed to start LXD host DNS service. -- Subject: Unit lxd-host-dns.service has failed
Could you kindly help me out of this mess? My lxd init is basically the defaults for the first cluster machine. I cant really imagine adding more machines to the cluster, if I can’t even run one host straight.
Hey, @simos, I think that this document about Ubuntu 18.04 might provide the solution to our problem.
It talks about his struggle to set up split-DNS on his Ubuntu Desktop. I didn’t follow the instructions (will do so when I get home) but just by looking at it sounds like what we need.
Hey @simos, I followed your instructions (creating a
lxd-host-dns.service file in /etc/systemd/system, etc), but now when that service is enabled, I get the problem that containers cannot translate a domain name (eg microsoft.com) to an IP address. And as soon as I
stop that service (the lxd-host-dns) with systemd, I can doing DNS requests again. Do you know why this is happening? And how it can be fixed?
@akarelas-pt so are you suggesting to revert to
ifupdown (or better yet
ifupdown2) inside the containers? I already noticed a number of shortcomings of
netplan.io since I encountered it (on a host with a fresh Ubuntu 18.04 install instead of an upgraded one).
No @Hvergelmir, everything I wrote in this forum thread is rubbish. That’s because I don’t know much about Linux Networking.
However THIS here suggestion (also by me) seems very good to me, and even though I don’t know enough about Linux Network to suggest it, it has worked excellently at my setups (both linux servers and desktops, and also both snap and apt-based installations): A way to resolve container.lxd from host, in all cases
Thanks for the response. Guess I have some more reading to do.
what should i do at this situation?
One way (that worked great for me) is to follow the very simple instructions at the top of this page: A way to resolve container.lxd from host, in all cases - you’ll need to undo anything you’ve done until now though.
Thank you @akarelas-pt
lxdbr0 address with reboot doesn’t change?
I don’t know, @alizowghi. But I’ve had this set-up on 2 PCs and it’s never changed on me.
It probably doesn’t change because if you type:
lxc network show lxdbr0 you get this, which maybe implies that the IP address is part of the network’s configuration:
config: ipv4.address: 10.100.12.34/24 ipv4.nat: "true" ipv6.address: faaa:faa:7777:bbbb::1/64 ipv6.nat: "true" description: "" name: lxdbr0 type: bridge used_by:
I tested your solution and till now is working very well.
I think that’s right.