I’m running into an issue where I’ve setup a type sriov network but instances are having NICs attached that are getting created with completely random hwaddr’s instead of expected addresses like 00:16:3e:xx:xx:xx where the xx is random and the vendor prefix for the ethernet address is for Xensource (00:16:3e).
Is this intentional? Or can this get fixed if not. I’d personally prefer sriov type networks use 00:16:3e:xx:xx:xx just like with bridged networking.
Container or VM? What version of Incus? What hardware vendor for the SR-IOV?
I know that some of that logic is pretty weird and vendor specific so will need to go look for your case specifically.
VM. Incus 6.0-LTS (just installed last week). Hardware is Intel x710-AT2 chip, 10G dual-port. Host is running Debian Trixie (testing) if that matters. Kernel 6.6.15-amd64.
Can you easily test if setting a hwaddr property on the nic entry does get it applied and working?
Adding this to the YAML configuration (in the GUI) before launching a new instance does not work, it still has random hwaddr.
devices:
eth0:
hwaddr: 00:16:3e:ae:4f:d0
network: sriov-thor
type: nic
eth1:
hwaddr: 00:16:3e:ae:4f:d1
network: sriov-thor
type: nic
eth2:
hwaddr: 00:16:3e:ae:4f:d2
network: sriov-thor
type: nic
eth3:
hwaddr: 00:16:3e:ae:4f:d3
network: sriov-thor
type: nic
volitle section when running has this:
config:
volatile.eth0.host_name: enp9s0f0v8
volatile.eth0.last_state.created: 'false'
volatile.eth0.last_state.hwaddr: 5a:3e:6f:f5:37:16
volatile.eth0.last_state.mtu: '1500'
volatile.eth0.last_state.pci.driver: iavf
volatile.eth0.last_state.vf.hwaddr: 5a:3e:6f:f5:37:16
volatile.eth0.last_state.vf.id: '8'
volatile.eth0.last_state.vf.parent: enp9s0f0
volatile.eth0.last_state.vf.spoofcheck: 'true'
volatile.eth0.last_state.vf.vlan: '0'
volatile.eth1.host_name: enp9s0f0v9
volatile.eth1.last_state.created: 'false'
volatile.eth1.last_state.hwaddr: 62:16:2c:c5:bd:23
volatile.eth1.last_state.mtu: '1500'
volatile.eth1.last_state.pci.driver: iavf
volatile.eth1.last_state.vf.hwaddr: 62:16:2c:c5:bd:23
volatile.eth1.last_state.vf.id: '9'
volatile.eth1.last_state.vf.parent: enp9s0f0
volatile.eth1.last_state.vf.spoofcheck: 'true'
volatile.eth1.last_state.vf.vlan: '0'
volatile.eth2.host_name: enp9s0f0v10
volatile.eth2.last_state.created: 'false'
volatile.eth2.last_state.hwaddr: d6:54:65:a2:8d:4d
volatile.eth2.last_state.mtu: '1500'
volatile.eth2.last_state.pci.driver: iavf
volatile.eth2.last_state.vf.hwaddr: d6:54:65:a2:8d:4d
volatile.eth2.last_state.vf.id: '10'
volatile.eth2.last_state.vf.parent: enp9s0f0
volatile.eth2.last_state.vf.spoofcheck: 'true'
volatile.eth2.last_state.vf.vlan: '0'
volatile.eth3.host_name: enp9s0f0v11
volatile.eth3.last_state.created: 'false'
volatile.eth3.last_state.hwaddr: 06:82:b0:13:1c:00
volatile.eth3.last_state.mtu: '1500'
volatile.eth3.last_state.pci.driver: iavf
volatile.eth3.last_state.vf.hwaddr: 06:82:b0:13:1c:00
volatile.eth3.last_state.vf.id: '11'
volatile.eth3.last_state.vf.parent: enp9s0f0
volatile.eth3.last_state.vf.spoofcheck: 'true'
volatile.eth3.last_state.vf.vlan: '0'
However… I stopped the instance, cleared out devices section (in the GUI) to:
devices: {}
and then from CLI did:
incus config device override Demo eth0 hwaddr=00:16:3e:ae:4f:d0
incus config device override Demo eth1 hwaddr=00:16:3e:ae:4f:d1
incus config device override Demo eth2 hwaddr=00:16:3e:ae:4f:d2
incus config device override Demo eth3 hwaddr=00:16:3e:ae:4f:d3
then started it back up, now it has the set hwaddr’s when I look at the interfaces inside the instance. The ‘volatile.ethX…’ show the same as before (in the GUI).
I don’t think this merge really fixed the underlying problem as I see it… I rebooted my server and when the SR-IOV VFs were initialized on start of the first VM, the VFs are still all assigned completely randow hwaddr’s…
ip link show dev enp9s0f0
...
vf 50 link/ether 72:b9:e0:42:f9:de brd ff:ff:ff:ff:ff:ff, spoof checking on, link-state auto, trust on
vf 51 link/ether ea:94:02:af:c8:ad brd ff:ff:ff:ff:ff:ff, spoof checking on, link-state auto, trust on
vf 52 link/ether a6:1a:01:95:94:44 brd ff:ff:ff:ff:ff:ff, spoof checking on, link-state auto, trust on
vf 53 link/ether 8e:90:e3:2b:71:b3 brd ff:ff:ff:ff:ff:ff, spoof checking on, link-state auto, trust on
vf 54 link/ether b6:16:b8:e1:b4:cf brd ff:ff:ff:ff:ff:ff, spoof checking on, link-state auto, trust on
vf 55 link/ether 52:c1:dd:2d:bb:8d brd ff:ff:ff:ff:ff:ff, spoof checking on, link-state auto, trust on
vf 56 link/ether fe:69:c2:eb:48:ff brd ff:ff:ff:ff:ff:ff, spoof checking on, link-state auto, trust on
vf 57 link/ether 02:9c:61:a9:4f:86 brd ff:ff:ff:ff:ff:ff, spoof checking on, link-state auto, trust on
vf 58 link/ether 3e:9a:5a:df:35:3f brd ff:ff:ff:ff:ff:ff, spoof checking on, link-state auto, trust on
vf 59 link/ether 76:c4:91:cb:40:96 brd ff:ff:ff:ff:ff:ff, spoof checking on, link-state auto, trust on
vf 60 link/ether 4a:f3:27:d7:2e:05 brd ff:ff:ff:ff:ff:ff, spoof checking on, link-state auto, trust on
vf 61 link/ether 66:ae:a8:ea:4d:f6 brd ff:ff:ff:ff:ff:ff, spoof checking on, link-state auto, trust on
vf 62 link/ether e6:8d:8b:aa:fd:58 brd ff:ff:ff:ff:ff:ff, spoof checking on, link-state auto, trust on
vf 63 link/ether 06:5e:0a:6a:fd:a3 brd ff:ff:ff:ff:ff:ff, spoof checking on, link-state auto, trust on
None are set with a vendor prefix of 00:16:3e as I would expect, so I still have to make sure to use the "incus device override … " to set hwaddr’s on each new VM after creation if I don’t want something completely random to be used and stored in the volatile state config.
I also have to manually turn on trust mode for the i40e driver (Intel x710 based card) to allow for the hwaddr’s to be changed prior to starting any VMs otherwise networking just doesn’t work. I get messages in the kernel log like:
i40e 0000:09:00.0: VF attempting to override administratively set MAC address, bring down and up the VF interface to resume normal operation
So to fix that, I do this and it’s happy to allow MAC changes on VM startup:
# for i in {0..63}; do ip link set dev enp9s0f0 vf $i spoof on trust on; done
All Incus does on first SR-IOV start is tell the kernel to enabled all the VFs.
It has no control on what the driver-assigned MAC will be at that stage.
When you actually start an instance and Incus takes control of one of the VFs, it will then record the current MAC, switch over to the new MAC address and then restore the old MAC when the instance gets stopped.
I see that it is picking a random 00:16:3e:xx:xx:xx for new instances. There’s just the “trust on” issue with the i40e driver and I’m working around that.