Multiple Bridge and IP allocation and Resolving Issues

LXD
1

Hi,

I have 2 bridges lxdbr0, lxdbr1 and I am experiencing some issues with ip-addressing and DNS resolution.
Hoping somebody can advise where I am going wrong.

lxc network list
+--------+----------+---------+-------------+---------+
|  NAME  |   TYPE   | MANAGED | DESCRIPTION | USED BY |
+--------+----------+---------+-------------+---------+
| eno1   | physical | NO      |             | 0       |
+--------+----------+---------+-------------+---------+
| lxdbr0 | bridge   | YES     |             | 12      |
+--------+----------+---------+-------------+---------+
| lxdbr1 | bridge   | YES     |             | 12      |
+--------+----------+---------+-------------+---------+
| virbr0 | bridge   | NO      |             | 0       |
+--------+----------+---------+-------------+---------+
| wlp3s0 | physical | NO      |             | 0       |
+--------+----------+---------+-------------+---------+

lxc network show lxdbr0
config:
  ipv4.address: 10.0.3.254/24
  ipv4.nat: "true"
  ipv6.address: none
description: ""
name: lxdbr0
type: bridge
used_by:
- /1.0/containers/tnt-ter-clr-0
- /1.0/containers/tnt-ter-clr-2
- /1.0/containers/tnt-ter-file-0
- /1.0/containers/tnt-ter-ha-0
- /1.0/containers/tnt-ter-infra-0
- /1.0/containers/tnt-ter-jss-0
- /1.0/containers/tnt-ter-mail-0
- /1.0/containers/tnt-ter-rds-0
- /1.0/containers/tnt-ter-rmq-0
- /1.0/containers/tnt-ter-sts-0
- /1.0/containers/tnt-ter-voip-0
- /1.0/containers/tnt-ter-www-0
managed: true
status: Created
locations:
- none

lxc network show lxdbr1
config:
  ipv4.address: 10.0.4.254/24
  ipv4.nat: "true"
  ipv6.address: none
description: ""
name: lxdbr1
type: bridge
used_by:
- /1.0/containers/tnt-vod-clr-1
- /1.0/containers/tnt-vod-clr-3
- /1.0/containers/tnt-vod-file-1
- /1.0/containers/tnt-vod-ha-1
- /1.0/containers/tnt-vod-infra-1
- /1.0/containers/tnt-vod-jss-1
- /1.0/containers/tnt-vod-mail-1
- /1.0/containers/tnt-vod-rds-1
- /1.0/containers/tnt-vod-rmq-1
- /1.0/containers/tnt-vod-sts-1
- /1.0/containers/tnt-vod-voip-1
- /1.0/containers/tnt-vod-www-1
managed: true
status: Created
locations:
- none

During creating the containers I specify the mac and ip address of each container using the following commands.

lxc config device set {{ item.hostname }} {{ item.iface }} ipv4.address {{ item.address }}
lxc config set {{ item.hostname }} volatile.{{ item.iface }}.hwaddr {{ item.mac }}

This results in each host having a file under /var/lib/lxd/networks/
where there is a folder for each bridge

sudo ls /var/lib/lxd/networks/
lxdbr0	lxdbr1

Each file under /var/lib/lxd/networks/lxdbr0/dnsmasq.hosts is correct and in the format 00:16:3e:4a:83:13,10.0.3.13,tnt-ter-clr-0

sudo ls /var/lib/lxd/networks/lxdbr0/dnsmasq.hosts
tnt-ter-clr-0	tnt-ter-ha-0	 tnt-ter-mail-0  tnt-ter-sts-0
tnt-ter-clr-2	tnt-ter-infra-0  tnt-ter-rds-0	 tnt-ter-voip-0
tnt-ter-file-0	tnt-ter-jss-0	 tnt-ter-rmq-0	 tnt-ter-www-0

and yet, I get problems with containers not getting their IP addresses.


lxc list
+-----------------+---------+------------------+------+------------+-----------+
|      NAME       |  STATE  |       IPV4       | IPV6 |    TYPE    | SNAPSHOTS |
+-----------------+---------+------------------+------+------------+-----------+
| tnt-ter-clr-0   | RUNNING | 10.0.3.13 (eth0) |      | PERSISTENT | 0         |
+-----------------+---------+------------------+------+------------+-----------+
| tnt-ter-clr-2   | RUNNING |                  |      | PERSISTENT | 0         |
+-----------------+---------+------------------+------+------------+-----------+
| tnt-ter-file-0  | RUNNING | 10.0.3.15 (eth0) |      | PERSISTENT | 0         |
+-----------------+---------+------------------+------+------------+-----------+
| tnt-ter-ha-0    | RUNNING |                  |      | PERSISTENT | 0         |
+-----------------+---------+------------------+------+------------+-----------+
| tnt-ter-infra-0 | RUNNING | 10.0.3.10 (eth0) |      | PERSISTENT | 0         |
+-----------------+---------+------------------+------+------------+-----------+
| tnt-ter-jss-0   | RUNNING |                  |      | PERSISTENT | 0         |
+-----------------+---------+------------------+------+------------+-----------+
| tnt-ter-mail-0  | RUNNING | 10.0.3.20 (eth0) |      | PERSISTENT | 0         |
+-----------------+---------+------------------+------+------------+-----------+
| tnt-ter-rds-0   | RUNNING | 10.0.3.17 (eth0) |      | PERSISTENT | 0         |
+-----------------+---------+------------------+------+------------+-----------+
| tnt-ter-rmq-0   | RUNNING |                  |      | PERSISTENT | 0         |
+-----------------+---------+------------------+------+------------+-----------+
| tnt-ter-sts-0   | RUNNING | 10.0.3.18 (eth0) |      | PERSISTENT | 0         |
+-----------------+---------+------------------+------+------------+-----------+
| tnt-ter-voip-0  | RUNNING |                  |      | PERSISTENT | 0         |
+-----------------+---------+------------------+------+------------+-----------+
| tnt-ter-www-0   | RUNNING |                  |      | PERSISTENT | 0         |
+-----------------+---------+------------------+------+------------+-----------+
| tnt-vod-clr-1   | RUNNING |                  |      | PERSISTENT | 0         |
+-----------------+---------+------------------+------+------------+-----------+
| tnt-vod-clr-3   | RUNNING |                  |      | PERSISTENT | 0         |
+-----------------+---------+------------------+------+------------+-----------+
| tnt-vod-file-1  | RUNNING |                  |      | PERSISTENT | 0         |
+-----------------+---------+------------------+------+------------+-----------+
| tnt-vod-ha-1    | RUNNING |                  |      | PERSISTENT | 0         |
+-----------------+---------+------------------+------+------------+-----------+
| tnt-vod-infra-1 | RUNNING | 10.0.4.10 (eth0) |      | PERSISTENT | 0         |
+-----------------+---------+------------------+------+------------+-----------+
| tnt-vod-jss-1   | RUNNING | 10.0.4.19 (eth0) |      | PERSISTENT | 0         |
+-----------------+---------+------------------+------+------------+-----------+
| tnt-vod-mail-1  | RUNNING | 10.0.4.20 (eth0) |      | PERSISTENT | 0         |
+-----------------+---------+------------------+------+------------+-----------+
| tnt-vod-rds-1   | RUNNING | 10.0.4.17 (eth0) |      | PERSISTENT | 0         |
+-----------------+---------+------------------+------+------------+-----------+
| tnt-vod-rmq-1   | RUNNING | 10.0.4.16 (eth0) |      | PERSISTENT | 0         |
+-----------------+---------+------------------+------+------------+-----------+
| tnt-vod-sts-1   | RUNNING | 10.0.4.18 (eth0) |      | PERSISTENT | 0         |
+-----------------+---------+------------------+------+------------+-----------+
| tnt-vod-voip-1  | RUNNING | 10.0.4.21 (eth0) |      | PERSISTENT | 0         |
+-----------------+---------+------------------+------+------------+-----------+
| tnt-vod-www-1   | RUNNING | 10.0.4.12 (eth0) |      | PERSISTENT | 0         |
+-----------------+---------+------------------+------+------------+-----------+

I also find that systemd-resolve is working very hard and resolution to domains from the containers is sometimes timing out.

  957 systemd+  20   0   71896   7388   5448 R  97.4  0.0 896:25.27 systemd-resolve      
18402 lxd       20   0   60104   2236   1828 R  35.4  0.0 288:01.41 dnsmasq              
18465 lxd       20   0   60104   2768   2312 R  31.5  0.0 293:12.54 dnsmasq
2

Looks like I am bumping into this issue with the exception that I do not have dnsmasq installed directly as a package. It is LXD that is providing the dnsmasq. But the general idea of the problem is very much the same.

3

So I found my problem. I am adding a service to systemd to enable .lxd domain resolution.

When I stop this service the problem is no longer presenting itself.

Any ideas how I can have .lxd domain resolution would be appreciated.

Here’s the content of my files.

The service

cat /lib/systemd/system/lxd-host-dns.service
[Unit]
Description=LXD host DNS service
After=lxd-containers.service

[Service]
Type=oneshot
ExecStart=/usr/local/bin/lxdhostdns_start.sh
RemainAfterExit=true
ExecStop=/usr/local/bin/lxdhostdns_stop.sh
StandardOutput=journal

[Install]
WantedBy=multi-user.target

The start script

cat /usr/local/bin/lxdhostdns_start.sh
#!/bin/sh
LXDINTERFACE=lxdbr0
LXDDOMAIN=lxd
LXDDNSIP=`ip addr show lxdbr0 | grep -Po 'inet \K[\d.]+'`

/usr/bin/systemd-resolve --interface ${LXDINTERFACE} \
                         --set-dns ${LXDDNSIP} \
                         --set-domain ${LXDDOMAIN}

LXDINTERFACE=lxdbr1
LXDDOMAIN=lxd
LXDDNSIP=`ip addr show lxdbr1 | grep -Po 'inet \K[\d.]+'`

/usr/bin/systemd-resolve --interface ${LXDINTERFACE} \
                         --set-dns ${LXDDNSIP} \
                         --set-domain ${LXDDOMAIN}

The stop script

cat /usr/local/bin/lxdhostdns_stop.sh
#!/bin/sh
LXDINTERFACE=lxdbr0

/usr/bin/systemd-resolve --interface ${LXDINTERFACE} --revert

LXDINTERFACE=lxdbr1

/usr/bin/systemd-resolve --interface ${LXDINTERFACE} --revert
4

This might help with .lxd resolution: Comment #14 : Bug #1571967 : Bugs : lxd package : Ubuntu