I have configured my bridge to have ipv4.dhcp.expiry=infinite so that IPs never change even when they’re shut down for long periods. However I have noticed that upon shutting down, certain virtual machines are removed from the lease file, and I don’t think leases are stored anywhere else so they’re presumably back in the pool of available IPs.
After incus start –all I have 39 entries in dnsmasq.leases, which is the number of lxc containers and qemu VMs that I have (I have no OCI container).
After incus stop –all I am down to 33 entries. It seems like all but one virtual machine entries are being removed. Perhaps they send a signal to dnsmasq to give up the lease during their shutdown procedure?
Is this the expected behaviour? And if so, is it only VMs that do that, or could a container conceivably give up its lease like that?
Is the only safe way of ensuring an IP never changes for a container to run incus config device override <name> eth0 ipv4.address=10.25.99.19 for each of them? I have confirmed that the dnsmasq.hosts entries for the VM persists in that situation (but not the lease).
I know that dnsmasq hashes the MAC so they’re pseudo-reserved, but I also know that collisions are frequent. Perhaps if I give it an entire /8 it would help? But regardless this isn’t a reliable assumption I’m willing to make.
That’s the safe way, and it’s what I do. (I think you could check your theory about the VMs giving up their leases on shutdown by using tcpdump)
I wasn’t aware of dnsmasq using the MAC address to seed the assignment. You shouldn’t ever get any actual collisions of course, since if the chosen address is already in another lease, a different one will be used.
However, I found one circumstance in which this happens: you clone a VM after it has created /etc/machineid. systemd-networkd uses the machineid to generate a DHCP Client ID, and dnsmasq by default prefers the Client ID over the MAC address. Therefore, multiple instances with same Client ID will all be assigned the same IP address. You can see these identifiers in the leases file.
You can fix this in dnsmasq by applying the following config to the network:
raw.dnsmasq: |
dhcp-ignore-clid
which makes dnsmasq only look at the MAC address - I do this as standard now. Or, you can delete /etc/machineid before publishing the image or cloning the instance. (I don’t know if incus has any built-in magic to do that for containers; I actually only came across this when cloning Proxmox VMs sitting on an incus managed network)
Aside: you might well ask why dnsmasq behaves like this by default. I think the idea was that if you connect the same machine via different network ports to the same subnet, you get assigned the same IP address. Great idea, except what happens if you connect on both wired and wireless simultaneously to the same subnet?
That’s the safe way, and it’s what I do. (I think you could check your theory about the VMs giving up their leases on shutdown by using tcpdump)
Thanks then I guess I will do that! Maybe make myself a little wrapper for incus create to automatically do it for me.
Tcpdump is good idea unfortunately I couldn’t get tcpdump it to work but I did spin up my own dnsmasq with logging and indeed I get a release request for VMs but not containers:
Jun 20 13:22:53 dnsmasq-dhcp[88452]: 1669255238 DHCPRELEASE(incusbr0) 10.25.99.166 10:66:6a:c6:a8:81
It’s nice that it doesn’t happen for containers, and I could probably configure VMs to never send it (possibly by forcefully killing the dhcp client early during shutdown, before it receives the signal itself?) but at this point it feels all brittle. So I’ll just have to get in the habit of always assigning ipv4.address!
dhcp-ignore-clid
Thank you I will add that to my configuration as well!
I’m a little bit new to containers and I find it confusing that the default use-case isn’t to have static IPs for every container. Do most people never address their containers by IP? How is communication usually done then? Things like reverse-proxy in front of VMs, or a web container talking to a database containers, or firewall rules.
Incus (well, dnsmasq really) creates DNS names for containers with active leases - these can be resolved by other containers on the same network (but not on other networks, or on the host itself). The default domain name is “incus” although you can change this. So “foo.incus” resolves to the address of container “foo”, as long as it’s running, or has been recently running (the lease is still there). If the container is down, it doesn’t resolve.
If you do want to refer to containers by IP, then setting ipv4.address is the way to do it. I guess incus could have included its own IP allocator, and assigned static IP addresses to every container it knows about (and released them back when the container is destroyed), but it was decided to use dnsmasq’s dynamic allocation.
I’d say that infinite DHCP lease times are not a good idea, because you could exhaust the pool - unless you manually remove leases for containers which no longer exist.