"Hey folks,
I’m kinda new to incus and LXC, and I’m facing a situation here that I’m not sure if it’s a bug or just a quirk of the network type I’m using. So, here’s the deal:
The setup on both machines (mac03 and mac04) is identical, just with the IP ending differently - one with .114 and the other with .118. Now, here’s the kicker: whenever I restart one of the machines (like ‘incus restart mac03’, for instance), the others lose connection to the network, and their ssh ECDSA host keys get changed, which is causing some complaints and worries from my customers, both on mac03 and mac04. Just to test, I spun up a container with Ubuntu 20.04 named ‘mac05’, and the situation was the same. When I used the ‘incus restart’ command to restart mac03, "mac04 and “mac05” lost connection to the network for about 2 or 3 seconds, and then when I tried to ssh into them, I got the ECDSA host key change warning for both mac04 and mac05. I’m pretty much at a loss on what to do to prevent this from happening, so I’m reaching out to you folks here in the forum for some help on fixing this issue. If you need any additional information, just let me know.
When you launch an instance with the macvlan networking, the instance by default gets the IP address from the external DHCP server. Your instance either gets a dynamic IP address or a static IP address. It depends on the configuration of the external DHCP server.
The external DHCP servers needs to be able to differentiate one instance from the other. Each instance normally has a random MAC address in the range of 00:16:3e:xx:xx:xx. But if you incus copy, etc, you might end up with two instances with the same MAC address.
Use this to figure out the MAC addresses for each instance:
$ incus config get myinstance1 volatile.eth0.hwaddr
00:16:3e:53:ea:10
As a first step, verify that each instance has a random but different MAC address. Also, configure your DHCP server to serve the same IP address per MAC address.
I don’t use dhcp anywhere. My datacenter has assigned me 32 ips, and all of these are statically associated with the “eno0” interface, which is used as the parent for the incus macvlan network. Inside vms, i do what i demonstrated in my post, configuring each vm to pick one of these static ips. BTW, is there a way for me to assign static mac addresses to the instances?
That may be an issue with macvlan in this case. Macvlan is a bit special in that it may vary in behavior between NIC vendors…
The behavior you’re describing would be somewhat consistent with something triggering a MAC change or a STP reset on the NIC (if connected to a switch that does STP).
Switch to using a bridge on the host system instead, that way Linux can handle STP once with the outside and then have extra virtual ports be added and removed without causing issues.
I found the solution. It was a stupid thing I was doing. An ip address conflict that I had caused by assigning all ips to “eno0” via /etc/network/interfaces. All I had to do was remove the ips from there and everything works like a charm. Thank you so much for your time and responses!