How to access container alex as alex.lxd from host?

akarelas-pt · June 14, 2018, 2:16pm

I remember that it is possible to set up DNS (some dns server somewhere) so that every LXD container of your host (eg one named alex) is accessible as .lxd

I remember it’s possible, because I have set it up on my old computer like that. But nomatter how much I search, I cannot find how to do it on my new LXD 3.0 (snap version).

Can you please show me how?
Can you please write this information in LXD’s documentation site? I don’t know for sure, but it sounds like very useful information to me.

Thank you.

simos · June 14, 2018, 7:59pm

Ubuntu 16.04 uses dnsmasq-base and it’s easy to configure an additional (prepended) DNS server, with LXD’s DNS server. You can find instructions on this if you search a bit on this forum.

However, Ubuntu 18.04 uses instead systemd-resolve as the DNS client. Therefore, you need to specify a la systemd the network configuration that you want for lxdbr0.

Here is how to do this,

akarelas-pt · June 18, 2018, 8:13am

Sadly, despite my tremendous efforts, I cannot get your permanent solution to work on my Ubuntu MATE 18.04 machine.

My .network file:
root@karjala-vmware:~# ls -al /etc/systemd/network/lxd.network
-rw-r–r-- 1 root root 60 Ιουν 18 11:08 /etc/systemd/network/lxd.network

root@karjala-vmware:~# cat /etc/systemd/network/lxd.network
[Match]
Name=lxdbr0

[Network]
DNS=10.123.194.1
Domains=lxd

However, even though I’ve reboot countless times, I get this, as if the .network files are not being read at all:

root@karjala-vmware:~# systemd-resolve --status lxdbr0
Link 3 (lxdbr0)
      Current Scopes: none
       LLMNR setting: yes
MulticastDNS setting: no
      DNSSEC setting: no
    DNSSEC supported: no

Do you know what it is I might be doing wrong?

simos · June 18, 2018, 8:19am

Do a

lxc network list

on the host to verify the network interface name.

Also

lxc network list-leases lxdbr0

to verify the containers are up and running.

akarelas-pt · June 18, 2018, 8:30am

The problem doesn’t seem to be there, I think:

root@karjala-vmware:/etc/systemd/network# lxc network list
+--------+----------+---------+-------------+---------+
|  NAME  |   TYPE   | MANAGED | DESCRIPTION | USED BY |
+--------+----------+---------+-------------+---------+
| ens33  | physical | NO      |             | 0       |
+--------+----------+---------+-------------+---------+
| lxdbr0 | bridge   | YES     |             | 1       |
+--------+----------+---------+-------------+---------+
root@karjala-vmware:/etc/systemd/network# lxc network list-leases lxdbr0
+----------+-------------------+---------------+---------+
| HOSTNAME |    MAC ADDRESS    |  IP ADDRESS   |  TYPE   |
+----------+-------------------+---------------+---------+
| alex     | 00:16:3e:6c:b0:f4 | 10.123.194.64 | DYNAMIC |
+----------+-------------------+---------------+---------+
root@karjala-vmware:/etc/systemd/network# host alex.lxd
Host alex.lxd not found: 3(NXDOMAIN)

akarelas-pt · June 18, 2018, 8:42am

I am getting some improvement, when I restart systemd-networkd. Rebooting is not enough, I have to explicitly type this afterwards, every time:

karjala@karjala-vmware:~$ sudo systemctl restart systemd-networkd
karjala@karjala-vmware:~$ systemd-resolve --status lxdbr0
Link 3 (lxdbr0)
      Current Scopes: none
       LLMNR setting: yes
MulticastDNS setting: no
      DNSSEC setting: no
    DNSSEC supported: no
         DNS Servers: 10.123.194.1
          DNS Domain: lxd

But still, I’m getting no dns resolution of alex.lxd:

karjala@karjala-vmware:~$ host alex.lxd
Host alex.lxd not found: 3(NXDOMAIN)
karjala@karjala-vmware:~$ sudo su -
root@karjala-vmware:~# systemctl restart systemd-resolved.service 
root@karjala-vmware:~# host alex.lxd
Host alex.lxd not found: 3(NXDOMAIN)

Could it be a bug of Ubuntu MATE?

akarelas-pt · June 18, 2018, 8:48am

Also before restarting systemd-networkd, the command dig @10.123.194.1 alex.lxd returns the correct result immediately. However after restarting it, the above dig command hangs until timeout. It looks like the problems are too many. I don’t know what’s wrong. The system is very fresh.

simos · June 18, 2018, 8:55am

Run on the host the following,

 host alex.lxd 10.123.194.1

This should work, so the problem would be with the first half of the setup on the host.

I have not tested my tutorial with the LXD snap, so there might be something there. Cannot think of something specific yet.
The most probably would be an issue with systemd-networkd. It does many things automatically so it either works fine or does not work at all. One issue with systemd-networkd is that if a DNS server does not respond early enough, then it’s not used later. But the configuration should be picked up.

Can you also run

sudo lsof -i -n | grep domain

to verify that indeed systemd-resolve is running?

akarelas-pt · June 18, 2018, 9:02am

karjala@karjala-vmware:~$ host alex.lxd 10.123.194.1
Using domain server:
Name: 10.123.194.1
Address: 10.123.194.1#53
Aliases: 

alex.lxd has address 10.123.194.64
karjala@karjala-vmware:~$ host alex.lxd
Host alex.lxd not found: 3(NXDOMAIN)
karjala@karjala-vmware:~$ 
karjala@karjala-vmware:~$ sudo lsof -i -n | grep domain
systemd-r  954 systemd-resolve   12u  IPv4  18216      0t0  UDP 127.0.0.53:domain 
systemd-r  954 systemd-resolve   13u  IPv4  18217      0t0  TCP 127.0.0.53:domain (LISTEN)
dnsmasq   1724          nobody    6u  IPv4  27277      0t0  UDP 10.123.194.1:domain 
dnsmasq   1724          nobody    7u  IPv4  27278      0t0  TCP 10.123.194.1:domain (LISTEN)
dnsmasq   1724          nobody   12u  IPv6  28723      0t0  UDP [fe80::9095:dfff:fe03:85dd]:domain 
dnsmasq   1724          nobody   13u  IPv6  28724      0t0  TCP [fe80::9095:dfff:fe03:85dd]:domain (LISTEN)

akarelas-pt · June 18, 2018, 9:12am

Until a more perfect solution is found, for now I just modified my /etc/systemd/resolved.conf file to look like this:

[Resolve]
DNS=10.123.194.1
Domains=lxd

…and it worked. I don’t know whether the Domains line is necessary.

simos · June 18, 2018, 9:54am

I have set up a LXD container with 18.04 (ubuntu:18.04), enabled nesting, set up the LXD snap in the container, and then created a nested container with 18.04 (ubuntu:18.04).

I did the configuration in /etc/systemd/network/lxd.network and then restarted the container.

I was able to ping the nested container, so it worked for me.
I cannot think of a reason why it should not work for your on the VMWare VM. There might be some (systemd?) race condition that did not show up in my case.

Regarding the network configuration for systemd (/etc/systemd/resolv.conf), see
https://www.freedesktop.org/software/systemd/man/resolved.conf.html

The Domains=lxd allows you to ping alex and it will work (because the DNS client will auto-append .lxd to make it ping alex.lxd.

The issue though with modifying /etc/systemd/resolved.conf is that (I think) you configure your whole system to use LXD’s DNS client for name resolutions.

It would be great if more people can try out the tutorial in order to figure out such cases and identify what troubleshooting to do.

Also, see https://github.com/systemd/systemd/issues/5755#issuecomment-296655750 for instructions on debugging.
I used this while writing my blog post in order to figure out what’s going on.

akarelas-pt · June 18, 2018, 11:45am

Hi. I created another VM, this time with ordinary Ubuntu 18.04 Desktop, and the same thing happened, as I had with Ubuntu MATE.

So it very well might not be the fault of my virtual machines. I’ll test your tutorial in a pure system, without VMware, when my new laptop arrives, in 2-3 weeks.

karjala · June 21, 2018, 6:00am

Sadly, I was right. I tested it on my new Ubuntu MATE laptop, with snap lxd, and host alex.lxd says host unknown.

simos · June 21, 2018, 9:21am

What’s the output of

systemd-resolve --status lxdbr0

As shown in my tutorial, you should get DNS in the Current scopes.
If you do not get DNS in the Current scopes, then the issue is how to get DNS to appear there.

simos · June 21, 2018, 1:46pm

I tried on a laptop (upgrade to 18.04, not clean install) as well and could not get it to work either. The Scope would not show DNS.
Apparently, things work better inside LXD containers .

Here are the steps for the name resolution of systemd-resolved
https://www.freedesktop.org/software/systemd/man/systemd-resolved.service.html
The issue is that systemd-resolved needs to be configured on the desktop to honor the per-link network configuration.

simos · June 21, 2018, 7:35pm

I tested with Ubuntu 18.04 in Virtualbox, and it worked as expected.
Therefore, there is some nuance somewhere there.

karjala · June 21, 2018, 8:08pm

In the past, I had set-up a dnsmasq on the host, just to forward .lxd requests to the LXD dnsmasq.

That’s also a (temporary) solution.

akarelas-pt · June 22, 2018, 6:06am

It worked?? Then that means that we are probably doing something different in our attempts to get it working. Because I also tried an Ubuntu under VMware and that hadn’t worked.

Let me try again…

akarelas-pt · June 22, 2018, 7:11am

@simos , when you say your tutorial worked on your VM (under VirtualBox), did your VM run LXD as a snap or as an apt package? Because if your VM was Ubuntu Server (and not Ubuntu Desktop), you would have tested it with apt (which is not the recommended way, I believe).

I tested the snap way a few more times, on new VMs. It didn’t work. Then I tested an Ubuntu Server (this comes with LXD pre-installed as an apt package) - your method worked there.

So what did you try? I’m interested in the snap way, because that’s the recommended way.

akarelas-pt · June 22, 2018, 7:35am

I tried uninstalling the pre-installed LXD that Ubuntu Server has by default, and installing LXD there from snap. Also did lxd init.

This time, when I tried your solution, the “systemd-resolve --status” command showed a DNS server in lxdbr0, but when I did lxc list, the IPv4 adderss of alex container was empty (alex only had an IPv6 address), and host alex.lxd didn’t return anything.

Whenever I removed the lxd.network file from /etc/systemd/network, alex would get an IPv4 after rebooting, but, of course, the host would still not be able to execute “host alex.lxd”

It seems Ubuntu Server and Ubuntu Desktop are behaving differently.

What was your setup when you said that it all worked?