Bug? Static IP configured in ubuntu-20.04 container, but not applied due to Netplan config

[pgoetz (Patrick Goetz)]

  OS: Arch Linux
  LXD: 4.19
  Default bridge: system br0 with static IP assignment
  Container creation command: `lxc launch images:ubuntu/20.04 samba-dc`

Earlier today @tomp showed me how to assign a static IP to a container, which I did:

# lxc config device override samba-dc eth0 ipv4.address=192.168.1.80

Here is the resulting config:

 [root@gecko ~]# lxc config show samba-dc
architecture: x86_64
config:
  image.architecture: amd64
  image.description: Ubuntu focal amd64 (20211018_07:42)
  image.os: Ubuntu
  image.release: focal
  image.serial: "20211018_07:42"
  image.type: squashfs
  image.variant: default
  volatile.base_image: a33719937baab258d01a998d443781f5eca73be877b5347f1cd9a6468a0b55b1
  volatile.eth0.host_name: veth84937c0c
  volatile.eth0.hwaddr: 00:16:3e:99:0f:4b
  volatile.idmap.base: "0"
  volatile.idmap.current: '[{"Isuid":true,"Isgid":false,"Hostid":100000,"Nsid":0,"Maprange":65536},{"Isuid":false,"Isgid":true,"Hostid":1000000,"Nsid":0,"Maprange":65536}]'
  volatile.idmap.next: '[{"Isuid":true,"Isgid":false,"Hostid":100000,"Nsid":0,"Maprange":65536},{"Isuid":false,"Isgid":true,"Hostid":1000000,"Nsid":0,"Maprange":65536}]'
  volatile.last_state.idmap: '[]'
  volatile.last_state.power: RUNNING
  volatile.uuid: c21d00d9-a2bd-44ab-bb86-270fa36f7ce7
devices:
  eth0:
    ipv4.address: 192.168.1.80
    name: eth0
    nictype: bridged
    parent: br0
    type: nic
ephemeral: false
profiles:
- default
stateful: false
description: ""
[root@gecko ~]# 

However, the container still comes up with a DHCP assigned address:

[root@gecko ~]# lxc list
+----------+---------+------+------+-----------+-----------+
|   NAME   |  STATE  | IPV4 | IPV6 |   TYPE    | SNAPSHOTS |
+----------+---------+------+------+-----------+-----------+
| samba-dc | STOPPED |      |      | CONTAINER | 0         |
+----------+---------+------+------+-----------+-----------+
[root@gecko ~]# lxc start samba-dc
[root@gecko ~]# lxc list
+----------+---------+----------------------+------+-----------+-----------+
|   NAME   |  STATE  |         IPV4         | IPV6 |   TYPE    | SNAPSHOTS |
+----------+---------+----------------------+------+-----------+-----------+
| samba-dc | RUNNING | 192.168.1.170 (eth0) |      | CONTAINER | 0         |
+----------+---------+----------------------+------+-----------+-----------+

Digging in to this I noticed that Netplan appears to be installed in the container image (why?!) and further that it contains a network configuration file:

[root@gecko ~]# lxc exec samba-dc -- bash
root@samba-dc:~# cat /etc/netplan/10-lxc.yaml 
network:
  version: 2
  ethernets:
    eth0:
      dhcp4: true
      dhcp-identifier: mac

Is this a bug or by design? I’ve never been a fan of Netplan, but given the elegance and simplicity of systemd-networkd, it’s truly an unnecessary middleman, in my opinion. But here’s where things get weird:

[root@gecko ~]# lxc exec samba-dc -- bash
root@samba-dc:~# apt remove netplan
Reading package lists... Done
Building dependency tree       
Reading state information... Done
Package 'netplan' is not installed, so not removed
0 upgraded, 0 newly installed, 0 to remove and 1 not upgraded.

If netplan isn’t installed, why does /etc/netplan/10-lxc.yaml exist and how is it being applied?

There are some netplan libs installed:

root@samba-dc:~# dpkg -l | grep netplan
ii  libnetplan0:amd64           0.103-0ubuntu5~20.04.1       amd64        YAML network configuration abstraction runtime library
ii  netplan.io                  0.103-0ubuntu5~20.04.1       amd64        YAML network configuration abstraction for various backends

So presumably these are responsible for the IP assgnment (there’s nothing in /etc/systemd/network). I’m at a loss for understanding how a DHCP assignment is superceding the static IP in the container configuration.

Further, if I take the bad actor out of the game:

root@samba-dc:~# cd /etc/netplan
root@samba-dc:/etc/netplan# mv 10-lxc.yaml 10-lxc.yaml-OFF

Then the container comes up with no IP address at all:

[root@gecko ~]# lxc stop samba-dc
[root@gecko ~]# lxc start samba-dc
[root@gecko ~]# lxc list
+----------+---------+------+------+-----------+-----------+
|   NAME   |  STATE  | IPV4 | IPV6 |   TYPE    | SNAPSHOTS |
+----------+---------+------+------+-----------+-----------+
| samba-dc | RUNNING |      |      | CONTAINER | 0         |
+----------+---------+------+------+-----------+-----------+

I thought I was starting to master this technology, but now I’m confused again. I believe it’s fair to call this a bug of some kind. If I reinstate the netplan yaml file, the DHCP-assigned IP address comes back:

[root@gecko ~]# lxc exec samba-dc -- bash
root@samba-dc:~# cd /etc/netplan
root@samba-dc:/etc/netplan# mv 10-lxc.yaml-OFF 10-lxc.yaml
root@samba-dc:/etc/netplan# exit
exit
[root@gecko ~]# lxc stop samba-dc
[root@gecko ~]# lxc start samba-dc
[root@gecko ~]# lxc list
+----------+---------+----------------------+------+-----------+-----------+
|   NAME   |  STATE  |         IPV4         | IPV6 |   TYPE    | SNAPSHOTS |
+----------+---------+----------------------+------+-----------+-----------+
| samba-dc | RUNNING | 192.168.1.170 (eth0) |      | CONTAINER | 0         |
+----------+---------+----------------------+------+-----------+-----------+

So clearly netplan is in charge of network configuration in this container and the LXD configuration is being ignored. What’s going on?

[tomp (Thomas Parrott)]

The package is called netplan.io, and the image ubuntu/20.04 isn’t built by the LXD team its built by the Ubuntu team. So LXD team have no control over the contents of it.

That being said the LXD built images:ubuntu/focal image also has netplan.io package as its the default in Ubuntu, and is what cloud-init uses to configure the network (via either systemd-networkd or NetworkManager).

[pgoetz (Patrick Goetz)]

OK, good to know. So I tried

apt remove netplan.io

and restarted the container, but I still don’t get an IP address:

[root@gecko ~]# lxc list
+----------+---------+------+------+-----------+-----------+
|   NAME   |  STATE  | IPV4 | IPV6 |   TYPE    | SNAPSHOTS |
+----------+---------+------+------+-----------+-----------+
| samba-dc | STOPPED |      |      | CONTAINER | 0         |
+----------+---------+------+------+-----------+-----------+
[root@gecko ~]# lxc start samba-dc
[root@gecko ~]# lxc list
+----------+---------+------+------+-----------+-----------+
|   NAME   |  STATE  | IPV4 | IPV6 |   TYPE    | SNAPSHOTS |
+----------+---------+------+------+-----------+-----------+
| samba-dc | RUNNING |      |      | CONTAINER | 0         |
+----------+---------+------+------+-----------+-----------+

The gap in my knowledge is how items from the configuration (like a static IP address) are applied to the container on start. But in any case, this isn’t working as advertised.

[tomp (Thomas Parrott)]

If you remove netplan.io then your container’s OS won’t request an IP via DHCP, which means LXD’s DHCP server won’t be able to assign the static allocation you specified. Which is why it doesn’t get an IP.

When you specify a static IP on the instance’s NIC device config, all it does is create a static DHCP assignment in LXD’s DHCP server associated to the NIC’s MAC address. This means you need to run a DHCP client inside the container for it to be applied.

If you want to use systemd-networkd that is fine, but you need to configure it use DHCP in order for it to access the LXD statically assigned IP address.

[tomp (Thomas Parrott)]

Are you still getting a different IP than your statically assigned one, is that the specific problem you’re having?

[pgoetz (Patrick Goetz)]

OK, I think I understand what is going on. My samba-dc container is configured to use the host br0 bridge, not lxdbr0, so it’s pulling an IP address from my DHCP server. The information I was missing was that you’re using a DHCP client to set up the IP address, and the configuration file is just setting up a static reservation for the internal LXD DHCP server. I think maybe I would have designed this differently.* This container absolutely must be public facing, so I’m going to have to work this out somehow.

• e.g. static assignments would not be handled by DHCP reservation because of this example – you don’t know what room the veth NIC is peering in to. Of course since not everyone is using systemd-networkd, this would create a lot of complications for LXD, which explains the current implementation.

Given the architecture, there seems to be only 2 options:

• Enable systemd-networkd and set a static IP assignment there (netplan would work for this as well of course).
• Set up an IP reservation on my own DHCP server for the container.

I guess either would work as long as the MAC address assigned to the veth interface doesn’t change.

[tomp (Thomas Parrott)]

Yes those are your options indeed

All our images by default try and configure their network via DHCP/SLAAC.

You could use cloud-init to configure the container’s config at create time rather than manually, but the end result is the same as manual configuration.

[pgoetz (Patrick Goetz)]

Is the architecture of LXD spelled out anywhere other than the source code? It would be extraordinarily helpful for people like me to know things like this in advance so that we can plan accordingly.

[tomp (Thomas Parrott)]

There’s some docs over at Networks | LXD

For example:

https://linuxcontainers.org/lxd/docs/master/networks#network-bridge

" As one of the possible network configuration types under LXD, LXD supports creating and managing network bridges. LXD bridges can leverage underlying native Linux bridges and Open vSwitch.

Creation and management of LXD bridges is performed via the lxc network command. A bridge created by LXD is by default “managed” which means that LXD also will additionally set up a local dnsmasq DHCP server and if desired also perform NAT for the bridge (this is the default.)

…

Additionally, LXD can utilize a pre-existing Linux bridge. In this case, the bridge does not need to be created via lxc network
"

One question I do have is that LXD shouldn’t have let you specify the ipv4.address on your NIC’s config as it should have known that you weren’t using a managed LXD bridge network.

Can you show the output of lxc network ls and lxc network show br0 please.

[tomp (Thomas Parrott)]

Ah yeah thats a bug.

[pgoetz (Patrick Goetz)]

See below:

[root@gecko ~]# lxc network ls
+--------+----------+---------+------+------+-------------+---------+
|  NAME  |   TYPE   | MANAGED | IPV4 | IPV6 | DESCRIPTION | USED BY |
+--------+----------+---------+------+------+-------------+---------+
| br0    | bridge   | NO      |      |      |             | 2       |
+--------+----------+---------+------+------+-------------+---------+
| enp3s0 | physical | NO      |      |      |             | 0       |
+--------+----------+---------+------+------+-------------+---------+
[root@gecko ~]#  lxc network show br0 
config: {}
description: ""
name: br0
type: bridge
used_by:
- /1.0/instances/samba-dc
- /1.0/profiles/default
managed: false
status: ""
locations: []
[root@gecko ~]#

[pgoetz (Patrick Goetz)]

This all works better than one could hope, as I have the flexibility to bend the containers to my needs. And I’m starting to see the benefit of using cloud-init. So, with cloud init I would invoke a script that does something like

NOTE: I spoke too soon and couldn’t get the following to work with the Ubuntu 20.04 image. See follow up comment below.

apt -y remove netplan.io
systemctl enable systemd-networkd
cat $foo > /etc/systemd/network/20-wired.network

where

$foo =
[Match]
Name=eth0

[Network]
Address=192.168.1.80/24
Gateway=192.168.1.1
DNS=192.168.1.1

[tomp (Thomas Parrott)]

This should avoid the confusion in the future:

NIC: Prevent use of static IPs on bridged NIC connected to unmanaged bridge

It will prevent using ipv{n}.address NIC settings when connecting to an unmanaged bridge.

[tomp (Thomas Parrott)]

Have a look at

How to get LXD containers get IP from the LAN with routed network

How to use the routed NIC type in LXD. I show how to setup individual LXD profiles with all the configuration in setting up the static IP address in the container. You can use the profile and directly launch a container with routed NIC.

Although it covers using the routed, rather than bridged NIC type, the tutorial also shows how to use cloud init to configure IPs manually.

[pgoetz (Patrick Goetz)]

Yes, I read this and many of Simos’ blog posts. I’m really not sure what Ubuntu is doing here, but I couldn’t get the systemd-networkd solution posted above to work. The container would start with no IP address again. If I update the netplan yaml files instead:

root@samba-dc:~# cat /etc/netplan/10-br0.yaml 
network:
  version: 2
  ethernets:
    eth0:
      dhcp4: no
      addresses:
      - 192.168.1.80/24
      gateway4: 192.168.1.1
      nameservers:
        addresses:
        - 192.168.1.1

Then this does work. I am curious why I can’t substitute systemd-networkd for netplan in the Ubuntu 20.04 container image, but don’t have time for that detour right now. I’ll try and test this the next time I do a bare metal Ubuntu 20.04 install. Meanwhile, my did you patch that bug quickly!