LXD bridge: new container will cause host ubuntu 18.04 network not work

1

Hi,

I am creating a network bridge for LXD container on ubuntu 18.04.

gpsemc@lxdtest:~$ ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen …
2: ens160: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel master br0 state UP group default qlen 1000
link/ether 00:50:56:bd:80:72 brd ff:ff:ff:ff:ff:ff
3: br0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
link/ether 3e:7e:2c:a4:d9:d0 brd ff:ff:ff:ff:ff:ff
inet 10.62.90.254/24 brd 10.62.90.255 scope global br0
valid_lft forever preferred_lft forever
inet6 fe80::3c7e:2cff:fea4:d9d0/64 scope link
valid_lft forever preferred_lft forever
5: vethFQGO7Y@if4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br0 state UP group default qlen 1000
…

Whenever I create a new container the network will be changed automatically

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
…
2: ens160: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel master br0 state UP group default qlen 1000
link/ether 00:50:56:bd:80:72 brd ff:ff:ff:ff:ff:ff
3: br0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
link/ether 3e:7e:2c:a4:d9:d0 brd ff:ff:ff:ff:ff:ff
inet 10.146.226.1/24 scope global br0
valid_lft forever preferred_lft forever
inet6 fd42:f4ee:39eb:4c7e:3c7e:2cff:fea4:d9d0/64 scope global dynamic mngtmpaddr noprefixroute
valid_lft 3425sec preferred_lft 3425sec
inet6 fd42:f4ee:39eb:4c7e::1/64 scope global
valid_lft forever preferred_lft forever
inet6 fe80::3c7e:2cff:fea4:d9d0/64 scope link
valid_lft forever preferred_lft forever
5: vethFQGO7Y@if4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br0 state UP group default qlen 1000
…

I have no idea how this IP comes from: 10.146.226.1/24. And the container created will be assigned with an IP:

root@test:~# ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
4: eth0@if5: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
link/ether 00:16:3e:28:f7:45 brd ff:ff:ff:ff:ff:ff link-netnsid 0
inet 10.146.226.108/24 brd 10.146.226.255 scope global dynamic eth0
valid_lft 2789sec preferred_lft 2789sec
inet6 fd42:f4ee:39eb:4c7e:216:3eff:fe28:f745/64 scope global deprecated dynamic mngtmpaddr noprefixroute
valid_lft 6361sec preferred_lft 0sec
inet6 fe80::216:3eff:fe28:f745/64 scope link
valid_lft forever preferred_lft forever

The bridge has issue. I need to run sudo netplan apply to recover the network on the host.

Below are the steps of how I create the host and bridge:

Host Setup:

sudo apt-get update    
sudo snap install lxd
lxd init   
lxd --version: 3.0.3

Refer to this article: https://www.cyberciti.biz/faq/install-lxd-pure-container-hypervisor-on-ubuntu-18-04-lts/

Refer to this article: https://linuxcontainers.org/lxd/getting-started-cli/

Create br0

sudo apt-get install bridge-utils

$ sudo nano /etc/sysctl.conf
Uncomment:

net.ipv4.ip_forward=1
Then, enable the new setting:

$ sudo sysctl -p

As our code hardcode to use a br0,

The rest: refer to https://openschoolsolutions.org/set-up-network-bridge-lxd/

Configure the NIC

Change the network configuration files under /etc/netplan/01-netcfg.yaml. It could be a difference name. Change it like this:

network:
version: 2
renderer: networkd

ethernets:
ens160:
dhcp4: no
bridges:
br0:
interfaces:
- ens160
addresses:
- 10.62.90.254/24
gateway4: 10.62.90.1
nameservers:
addresses:
- 10.254.174.10
dhcp4: no

sudo netplan generate
sudo netplan --debug apply

Refer to this page: https://fabianlee.org/2019/04/01/kvm-creating-a-bridged-network-with-netplan-on-ubuntu-bionic/

Generate the certificate

Refer to this section https://linuxcontainers.org/lxd/docs/master/security → Adding a remote with TLS client certificate authentication

lxc remote add ubuntu1 https://cloud-images.ubuntu.com/releases [This command will fail, but it will generate the certificate anyway]

Then it will generate a client certificate under ~/.config/lxc

Interacting with API

lxc config set core.https_address “[::]:8443”
lxc config set core.trust_password gpsemc
curl --unix-socket /var/lib/lxd/unix.socket s/

sudo apt install jq

curl -s --unix-socket /var/lib/lxd/unix.socket s/ | jq .
curl -s -k --cert ~/.config/lxc/client.crt --key ~/.config/lxc/client.key https://127.0.0.1:8443/1.0 | jq .metadata.auth
curl -s -k --cert ~/.config/lxc/client.crt --key ~/.config/lxc/client.key https://127.0.0.1:8443/1.0/certificates -X POST -d ‘{“type”: “client”, “password”: “gpsemc”}’ | jq .
curl -s -k --cert ~/.config/lxc/client.crt --key ~/.config/lxc/client.key https://127.0.0.1:8443/1.0 | jq .metadata.auth

Refer to this article: https://stgraber.org/2016/04/18/lxd-api-direct-interaction/

2

Anyone has clue? The similar configuration works on ubuntu 16.04

3

Can you show the output of lxc network list on the host please?

4

LXD looks for “lxcbr0” for doing any container operations in 18.04.
LXD should create “lxcbr0” by default on 18.04
Just change “br0” to “lxcbr0” in your netplan configuration YAML file.

5

gpsemc@lxdtest:~$ lxc network list
±-------±---------±--------±------------±--------+
| NAME | TYPE | MANAGED | DESCRIPTION | USED BY |
±-------±---------±--------±------------±--------+
| br0 | bridge | YES | | 0 |
±-------±---------±--------±------------±--------+
| ens160 | physical | NO | | 0 |
±-------±---------±--------±------------±--------+

6

So the br0 interface is showing as “managed” in the lxc network list, this means that LXD will bring the interface up and change its configuration.

If you do lxc network show br0 you should see what its configured address is.

You should not have LXD managed networks also managed by another system (like netplan), otherwise the two systems will conflict.

If your intention is to use br0 to join the host’s physical network interface to it as a bridge, and then connect your containers to it as well, then you don’t need LXD to manage it and so you should remove it as a managed network from LXD.

Then you can get your containers to connect to it (when managed by netplan) by simply setting the container’s bridged NIC parent to the interface.

1 Like