Hello guys, I have a fresh new LXD 3.13 cluster with a FAN network.
DNS resolution works fine on the containers, I would like to enable DNS for the (3) host machines.
Any pointers? Thanks a lot!
I’m on Ubuntu 18.04: Is netplan the way to go?
It depends. Take a look at
https://lxd.readthedocs.io/en/latest/faq/#networking-issues
Thank you @paulo_bruck. Could you please elaborate?
The link you provided reads " Do not use systemd-networkd with netplan and bridges based on vlans"
I have configured 01-netcfg.yaml so I’m using netplan and vlans are created for each container, right?
But how do I know if I’m using systemd-networkd?
REALLY WEIRD BEHAVIOR!!!
I created a 01-netcfg.yaml by copying the contents of 50-cloud-init.yaml and then adding my own config:
This is 50-cloud-init.yaml:
# This file is generated from information provided by
# the datasource. Changes to it will not persist across an instance.
# To disable cloud-init's network configuration capabilities, write a file
# /etc/cloud/cloud.cfg.d/99-disable-network-config.cfg with the following:
# network: {config: disabled}
network:
version: 2
ethernets:
eth0:
addresses:
- 165.22.78.66/20
- 10.19.0.9/16
gateway4: 165.22.64.1
match:
macaddress: b6:02:4b:29:69:c4
nameservers:
addresses:
- 67.207.67.2
- 67.207.67.3
search: []
set-name: eth0
eth1:
addresses:
- 10.135.50.133/16
match:
macaddress: b6:58:e5:10:28:10
nameservers:
addresses:
- 67.207.67.2
- 67.207.67.3
search: []
set-name: eth1
This is the new 01-netcfg.yaml with 2 changes (additions):
> network:
> version: 2
> ethernets:
> eth0:
> addresses:
> - 165.22.78.66/20
> - 10.19.0.9/16
> gateway4: 165.22.64.1
> match:
> macaddress: b6:02:4b:29:69:c4
> nameservers:
> addresses:
> - 67.207.67.2
> - 67.207.67.3
> search: []
> set-name: eth0
> eth1:
> addresses:
> - 10.135.50.133/16
> match:
> macaddress: b6:58:e5:10:28:10
> nameservers:
> addresses:
> - 240.50.133.1 # (this is the IP of the lxdfan0 interface)
> - 67.207.67.2
> - 67.207.67.3
> search:
> - lxd # added lxd to search
> set-name: eth1
After issuing: netplan apply
I am able to resolve c1.lxd as well as c1
However this is true for one or two attempts, after that I get:
Name or service not known
or
Temporary failure in name resolution
If I re-issue netplan apply I am able to resolve again, for one or two times, then back to the errors…
Rebooting the server results in the same behavior, DNS on the host works for one attempt or two, then back to the errors.
In all cases pinging external domains works on the host.
In all cases pinging containers from within containers still works fine, both c1.lxd and just c1 work
This is netplan --debug aply:
NOTE: See how my extra config is not in “DEBUG:Merged config:”, but then… why does it work the first time?
root@host01:~# sudo netplan --debug apply ** (generate:5671): DEBUG: 22:30:53.719: Processing input file /etc/netplan/01-netcfg.yaml.. ** (generate:5671): DEBUG: 22:30:53.719: starting new processing pass ** (generate:5671): DEBUG: 22:30:53.720: Processing input file /etc/netplan/50-cloud-init.yaml.. ** (generate:5671): DEBUG: 22:30:53.720: starting new processing pass ** (generate:5671): DEBUG: 22:30:53.720: eth1: setting default backend to 1 ** (generate:5671): DEBUG: 22:30:53.720: Configuration is valid ** (generate:5671): DEBUG: 22:30:53.720: eth0: setting default backend to 1 ** (generate:5671): DEBUG: 22:30:53.720: Configuration is valid ** (generate:5671): DEBUG: 22:30:53.720: Generating output files.. ** (generate:5671): DEBUG: 22:30:53.721: NetworkManager: definition eth0 is not for us (backend 1) ** (generate:5671): DEBUG: 22:30:53.721: NetworkManager: definition eth1 is not for us (backend 1) DEBUG:netplan generated networkd configuration changed, restarting networkd DEBUG:no netplan generated NM configuration exists DEBUG:eth0 not found in {} DEBUG:eth1 not found in {'eth0': {'addresses': ['165.22.78.66/20', '10.19.0.9/16'], 'gateway4': '165.22.64.1', 'match': {'macaddress': 'b6:02:4b:29:69:c4'}, 'nameservers': {'addresses': ['240.50.133.1', '67.207.67.2', '67.207.67.3'], 'search': ['lxd']}, 'set-name': 'eth0'}} DEBUG:eth0 exists in {'eth0': {'addresses': ['165.22.78.66/20', '10.19.0.9/16'], 'gateway4': '165.22.64.1', 'match': {'macaddress': 'b6:02:4b:29:69:c4'}, 'nameservers': {'addresses': ['240.50.133.1', '67.207.67.2', '67.207.67.3'], 'search': ['lxd']}, 'set-name': 'eth0'}, 'eth1': {'addresses': ['10.135.50.133/16'], 'match': {'macaddress': 'b6:58:e5:10:28:10'}, 'nameservers': {'addresses': ['240.50.133.1', '67.207.67.2', '67.207.67.3'], 'search': ['lxd']}, 'set-name': 'eth1'}} DEBUG:eth1 exists in {'eth0': {'addresses': ['165.22.78.66/20', '10.19.0.9/16'], 'gateway4': '165.22.64.1', 'match': {'macaddress': 'b6:02:4b:29:69:c4'}, 'nameservers': {'addresses': ['67.207.67.2', '67.207.67.3'], 'search': []}, 'set-name': 'eth0'}, 'eth1': {'addresses': ['10.135.50.133/16'], 'match': {'macaddress': 'b6:58:e5:10:28:10'}, 'nameservers': {'addresses': ['240.50.133.1', '67.207.67.2', '67.207.67.3'], 'search': ['lxd']}, 'set-name': 'eth1'}} DEBUG:Merged config: network: bonds: {} bridges: {} ethernets: eth0: addresses: - 165.22.78.66/20 - 10.19.0.9/16 gateway4: 165.22.64.1 match: macaddress: b6:02:4b:29:69:c4 nameservers: addresses: - 67.207.67.2 - 67.207.67.3 search: [] set-name: eth0 eth1: addresses: - 10.135.50.133/16 match: macaddress: b6:58:e5:10:28:10 nameservers: addresses: - 67.207.67.2 - 67.207.67.3 search: [] set-name: eth1 vlans: {} wifis: {} DEBUG:Skipping non-physical interface: lo DEBUG:device eth0 operstate is up, not changing DEBUG:device eth1 operstate is up, not changing DEBUG:Skipping non-physical interface: lxdfan0 DEBUG:Skipping non-physical interface: lxdfan0-mtu DEBUG:Skipping non-physical interface: lxdfan0-fan DEBUG:Skipping non-physical interface: vethBBU05C DEBUG:Skipping non-physical interface: vethGJTBEE DEBUG:{} DEBUG:netplan triggering .link rules for lo DEBUG:netplan triggering .link rules for eth0 DEBUG:netplan triggering .link rules for eth1 DEBUG:netplan triggering .link rules for lxdfan0 DEBUG:netplan triggering .link rules for lxdfan0-mtu DEBUG:netplan triggering .link rules for lxdfan0-fan DEBUG:netplan triggering .link rules for vethBBU05C DEBUG:netplan triggering .link rules for vethGJTBEE
Note how here my new config seems to be added as it should:
root@host01:~# systemd-resolve --status
Global
DNSSEC NTA: 10.in-addr.arpa
16.172.in-addr.arpa
168.192.in-addr.arpa
17.172.in-addr.arpa
18.172.in-addr.arpa
19.172.in-addr.arpa
20.172.in-addr.arpa
21.172.in-addr.arpa
22.172.in-addr.arpa
23.172.in-addr.arpa
24.172.in-addr.arpa
25.172.in-addr.arpa
26.172.in-addr.arpa
27.172.in-addr.arpa
28.172.in-addr.arpa
29.172.in-addr.arpa
30.172.in-addr.arpa
31.172.in-addr.arpa
corp
d.f.ip6.arpa
home
internal
intranet
lan
local
private
test
Link 10 (vethGJTBEE)
Current Scopes: none
LLMNR setting: yes
MulticastDNS setting: no
DNSSEC setting: no
DNSSEC supported: no
Link 8 (vethBBU05C)
Current Scopes: none
LLMNR setting: yes
MulticastDNS setting: no
DNSSEC setting: no
DNSSEC supported: no
Link 6 (lxdfan0-fan)
Current Scopes: none
LLMNR setting: yes
MulticastDNS setting: no
DNSSEC setting: no
DNSSEC supported: no
Link 5 (lxdfan0-mtu)
Current Scopes: none
LLMNR setting: yes
MulticastDNS setting: no
DNSSEC setting: no
DNSSEC supported: no
Link 4 (lxdfan0)
Current Scopes: none
LLMNR setting: yes
MulticastDNS setting: no
DNSSEC setting: no
DNSSEC supported: no
Link 3 (eth1)
Current Scopes: DNS
LLMNR setting: yes
MulticastDNS setting: no
DNSSEC setting: no
DNSSEC supported: no
DNS Servers: 240.50.133.1
67.207.67.2
67.207.67.3
DNS Domain: lxd
Link 2 (eth0)
Current Scopes: DNS
LLMNR setting: yes
MulticastDNS setting: no
DNSSEC setting: no
DNSSEC supported: no
DNS Servers: 240.50.133.1
67.207.67.2
67.207.67.3
DNS Domain: lxdHi Cadilla
I 'm still learning LXD, but here is what they talk about netplan:( see bold part)
Do not use systemd-networkd with netplan and bridges based on vlans
At time of writing (2019-03-05), netplan can not assign a random MAC address to a bridge attached to a vlan. It always picks the same MAC address, which causes layer2 issues when you have more than one machine on the same network segment. It also has difficultly creating multiple bridges. Make sure you use network-manager instead. An example config is below, with a management address of 10.61.0.25, and VLAN102 being used for client traffic.
network:
version: 2
renderer: **NetworkManager**
When you use netplan default render is **network**........
Hope it helps 8)
Setting renderer: NetworkManager
Completely breaks DNS in my setup…
@karjala Your solution is working fine in my environment, can you please explain the logic behind it and how it fits / works in combination with netplan and systemd-networkd? Thank you!
Step1: create a new file named /etc/systemd/network/lxd.network with these contents:
[Match]
Name=lxdbr0
[Network]
DNS=10.121.179.1
Domains=~lxd
[Address]
Address=10.121.179.1/24
Gateway=10.121.179.1
(replace 10.121.179.1 with the address of your lxdbr0 bridge)
Step 2: do this: sudo systemctl enable systemd-networkd
Step 3: reboot your machine.
Finished.
I kept reading everything I could and I figured out that @rudiservo was the initial author of this solution.
Hey @Yosu_Cadilla
I do not know about FAN and cluster, haven’t got that far, sorry.
Here is what I know.
In 18.04 NetPlan is the default manager, somehow WOL flag is not working in netplan
18.04 has the systemd-resolved, everytime you do try to access a name that does not exists resolved default is query to 8.8.8.8 dns server, so it bypasses your current dns config for that name your trying to get, the only way to get it back is to restart resolved service.
One of the solutions is to ditch resolved and turn on dsnmasq.
The other one is to have something that adds those containers into your hosts file.
In terms of clustering my only rant is things are more complicated then you expect and it seems that there is no turn key solution to ease up management, trying openstack and I am quite disapointed…
Thanks for your insight @rudiservo, I tried to ditch systemd-resolved, but DNS stopped working cluster-wide.
@stgraber told me on a different thread that he added some glue for DNS to work out of the box on a clustered LXD with a FAN, so as I’m really far from being a networking expert, I won’t try tinkering with that and I will stick to your solution for now.
The fact that the host machines “are not part of the cluster” does, in fact, simplify things a lot in this regard.
What I am trying to build is a way to manage the cluster from one node automatically, DNS is not really needed, however it makes life better for me and the devs. In case someone else wants to do the same and this DNS patch doesn’t do the job, here’s the other thing I am working on that would allow doing the same:
In this post, @simos explains hot to issue commands on the host from a container, hence allowing the manager to be isolated and portable, I haven’t tested it yet, however, since DNS works out of the box from the containers, I guess it’s a viable alternative to patching DNS.