Multiple Leases on lxdbr0

Hello,

I am encountering a strange issue where I have set ipv4.address on current profile for a virtual machine but it seems its still pulling some address dynamically.

$ lxc network list-leases lxdbr0
±---------±------------------±--------------±--------+
| HOSTNAME | MAC ADDRESS | IP ADDRESS | TYPE |
±---------±------------------±--------------±--------+
| kubeadm1 | 00:16:3e:42:c0:82 | 10.29.146.100 | STATIC |
±---------±------------------±--------------±--------+
| kubeadm1 | 00:16:3e:42:c0:82 | 10.29.146.110 | DYNAMIC |
±---------±------------------±--------------±--------+
| kubeadm2 | 00:16:3e:1d:e3:0f | 10.29.146.101 | STATIC |
±---------±------------------±--------------±--------+
| kubeadm2 | 00:16:3e:1d:e3:0f | 10.29.146.165 | DYNAMIC |
±---------±------------------±--------------±--------+
| kubeadm3 | 00:16:3e:54:ea:57 | 10.29.146.102 | STATIC |
±---------±------------------±--------------±--------+
| kubeadm3 | 00:16:3e:54:ea:57 | 10.29.146.70 | DYNAMIC |
±---------±------------------±--------------±--------+

Current profile

$ lxc profile show kubeadm2
config:
limits.cpu: “2”
limits.memory: 2GB
description: Default LXD profile
devices:
eth0:
ipv4.address: 10.29.146.101
name: eth0
network: lxdbr0
type: nic
root:
path: /
pool: storpool1
type: disk
name: kubeadm2
used_by:

  • /1.0/instances/kubeadm2

Instance:

~$ lxc config show kubeadm2
architecture: x86_64
config:
image.architecture: amd64
image.description: ubuntu 20.04 LTS amd64 (release) (20210510)
image.label: release
image.os: ubuntu
image.release: focal
image.serial: “20210510”
image.type: disk-kvm.img
image.version: “20.04”
volatile.base_image: f217de8a505d919f42cac6a898ac66fdf582b8ccc092d73189b46101615f3612
volatile.eth0.host_name: tap19c1ca33
volatile.eth0.hwaddr: 00:16:3e:1d:e3:0f
volatile.last_state.power: RUNNING
volatile.uuid: 47cb0965-feda-4bb8-837c-646863773ac7
devices: {}
ephemeral: false
profiles:

  • kubeadm2
    stateful: false
    description: “”

Does anyone know how I can clear that dynamic address? Thanks in advance

The old lease should expire after a time and the new static lease should take over.

You can speed this up by stopping dnsmasq, manually editing the leases file and then reloading LXD (which will start dnsmasq again).

Thanks @tomp I already rebooted it and can still see this on lxd network list-leases. One thing to note is that the ipv4.address where already specified in the profile prior to doing init and starting the VM. It seems to me that it just suddenly ignored the config in the profile. I will give it a try tomorrow once I have access to it and update here. I’m using 4.14 version by the way.

Can you show me the contents of the leases file?

You can see the file being used in the dnsmasq args:

ps aux | grep dnsmasq

Also, out of interest, why are you using multiple profiles for the statically assigned IPs, rather than adding the NIC devices with static IP directly to the container?

Hey @tomp , looks okay now. No idea what happened but after bringing up the server again /var/snap/lxd/common/lxd/networks/lxdbr0/dnsmasq.leases is empty. Starting the VM in LXD, I can now see the static leases properly. Also, these are not containers but VMs. I cannot reproduce this on containers.

$ lxc network list-leases lxdbr0
±---------±------------------±--------------±-------+
| HOSTNAME | MAC ADDRESS | IP ADDRESS | TYPE |
±---------±------------------±--------------±-------+
| kubeadm1 | 00:16:3e:42:c0:82 | 10.29.146.100 | STATIC |
±---------±------------------±--------------±-------+
| kubeadm2 | 00:16:3e:1d:e3:0f | 10.29.146.101 | STATIC |
±---------±------------------±--------------±-------+
| kubeadm3 | 00:16:3e:54:ea:57 | 10.29.146.102 | STATIC |
±---------±------------------±--------------±-------+

Regarding your question, I used to copy out the VMs to another LXD to conserve storage and copy back in when I need it. Profiles help me to persist the ip and the limits prior to starting after copy. Or maybe I don’t know an easier way as I just started learning/using. By the way, maybe a bit off topic from my original one. I have tried and confirm in the documentation that ipv4.address is not part of parameter for macvlan. I wonder whether there is a reason why its ommited? Thanks in advance.

You can do this:

lxc init images:ubuntu/focal v1 --vm
lxc config device override v1 eth0 ipv4.address=n.n.n.n
lxc start v1

This will copy the profile’s eth0 config into the VM’s own config and then update its ipv4.address setting, so that has its own config. This avoids needing to create a profile for each instance.

I was wondering if perhaps the use of profiles was causing this issue as I was not able to reproduce the issue. I’ll try with VMs.

The macvlan NIC type doesn’t use LXD’s dnsmasq process (as macvlan NICs cannot communicate with the LXD host) and so there is no way to setup a static DHCP lease via the ipv4.address setting.

Here’s my steps to try and reproduce (but didn’t succeed):

# Empty leases file.
sudo cat /var/snap/lxd/common/lxd/networks/lxdbr0/dnsmasq.leases
duid 00:01:00:01:28:3f:77:17:98:ee:cb:c2:1b:4e

# Create a VM and start.
lxc launch images:ubuntu/focal v1 --vm

# Check leases file contains new dnyamic lease.
sudo cat /var/snap/lxd/common/lxd/networks/lxdbr0/dnsmasq.leases
1621962881 00:16:3e:37:93:c4 10.130.212.5 v1 01:00:16:3e:37:93:c4
duid 00:01:00:01:28:3f:77:17:98:ee:cb:c2:1b:4e

# Stop VM and assign different static IP.
lxc stop v1
lxc config device override v1 eth0 ipv4.address=10.130.212.10

# Check old lease still present.
sudo cat /var/snap/lxd/common/lxd/networks/lxdbr0/dnsmasq.leases
1621962881 00:16:3e:37:93:c4 10.130.212.5 v1 01:00:16:3e:37:93:c4
duid 00:01:00:01:28:3f:77:17:98:ee:cb:c2:1b:4e

# Start VM and check new static lease is applied.
lxc ls v1
+------+---------+------------------------+------------------------------------------------+-----------------+-----------+
| NAME |  STATE  |          IPV4          |                      IPV6                      |      TYPE       | SNAPSHOTS |
+------+---------+------------------------+------------------------------------------------+-----------------+-----------+
| v1   | RUNNING | 10.130.212.10 (enp5s0) | fd42:9e9d:2fd:86d5:216:3eff:fe37:93c4 (enp5s0) | VIRTUAL-MACHINE | 0         |
+------+---------+------------------------+------------------------------------------------+-----------------+-----------+

# Check new lease is written and old one removed.
sudo cat /var/snap/lxd/common/lxd/networks/lxdbr0/dnsmasq.leases
1621963061 00:16:3e:37:93:c4 10.130.212.10 v1 01:00:16:3e:37:93:c4
duid 00:01:00:01:28:3f:77:17:98:ee:cb:c2:1b:4e

Thanks @tomp. I will let you know if I manage to reproduce it. So far, I encountered it only once. I was thinking as well if I may have passed a wrong interface name cause this VM seems to change interface name. from eth0 to enp5s0 e.g.

±--------------------±--------±-----------------------±-----±----------------±----------+
| kubeadm1 | RUNNING | 10.29.146.100 (eth0) | | VIRTUAL-MACHINE | 1 |
±--------------------±--------±-----------------------±-----±----------------±----------+
| kubeadm2 | RUNNING | 10.29.146.101 (eth0) | | VIRTUAL-MACHINE | 1 |
±--------------------±--------±-----------------------±-----±----------------±----------+
| kubeadm3 | RUNNING | 10.29.146.102 (enp5s0) | | VIRTUAL-MACHINE | 1 |
±--------------------±--------±-----------------------±-----±----------------±----------+

Eventually, having 3 interfaces (main, docker and overlay)

±--------------------±--------±-----------------------±-----±----------------±----------+
| kubeadm1 | RUNNING | 192.168.107.64 (tunl0) | | VIRTUAL-MACHINE | 1 |
| | | 172.17.0.1 (docker0) | | | |
| | | 10.29.146.100 (enp5s0) | | | |
±--------------------±--------±-----------------------±-----±----------------±----------+

Thats expected I’m afraid. We have no way to pass the specified name property (as opposed to the device name) into the VM (like we can with containers). So the actual interface name is derived from the virtual PCI slot. The PCI slot is static and is derived from the order of the VM’s devices (NICs go first, followed by disk and then other devices, ordered by device name).

Before the lxd-agent has started inside the VM, LXD tries to ‘guess’ what the IPs will be (by looking at the dnsmasq leases file and the host’s ARP cache and trying to match on the NIC’s MAC address), and uses the device’s name from the config to give them context. This is why it changes once the VM has started and the lxd-agent is running inside the VM. This is useful when running VMs that don’t have an lxd-agent (like Windows right now) as often the IPs are correct.

But this doesn’t affect the output of lxc network list-leases lxdbr0

Thank you so much for the patience and help. I really like this community cause people are very helpful and supportive. I am enjoying LXD and helped me a lot on my work. Thanks.

1 Like