I have a 3 host incus cluster, all hosts running incus 6.6 on debian 12, identical hardware across the board. I set migration.stateful=true on my profiles I use for virtual machines and after rebooting the VMs it seems incus is only launching qemu processes with all the CPU flags set on one of the hosts in the cluster, the other two hosts are only launching VMs with -cpu kvm64.
I tried setting instances.vm.cpu.x86_64.baseline=kvm64 instances.vm.cpu.x86_64.flags=auto on the default cluster group as outlined in the 6.4 release announcement but it seems only instances.vm.cpu.x86_64.baseline=kvm64 is being populated. I’m able to work around this by manually populating all the computed flags from the working server in instances.vm.cpu.x86_64.flags.
I’m not sure if this is a bug, intended behavior, or if there’s something I’m doing wrong.
Thanks for the response and example. I guess the specific question I have is if the following is expected behavior with a cluster group with default configuration:
Only instance test2 launched with additional cpu flags set, the other two just launched with kvm64. The host test2 launched on is the current database-leader, not sure if that has any bearing on how the flags are decided:
Might be worth checking that all your servers have every server listed in /var/cache/incus/resources/ and that those .yaml do in fact include the correct CPU flags?
And the contents seemed all over the place (side note, these files end in .yaml, but the content is actually json):
david@foobar ~ $ for host_name in incus{1..3}; do echo -e "\nhost ${host_name}:"; echo -n " cpu flags count from API: "; ssh ${host_name} "sudo incus query /1.0/resources | jq .cpu.sockets[0].cores[0].flags | wc -l"; echo " cached cpu flags count per member:"; for file_name in incus{1..3}.yaml; do echo -n " ${file_name}: "; ssh ${host_name} "sudo cat /var/cache/incus/resources/${file_name} | jq .cpu.sockets[0].cores[0].flags | wc -l"; done; done
host incus1:
cpu flags count from API: 139
cached cpu flags count per member:
incus1.yaml: 141
incus2.yaml: 1
incus3.yaml: 1
host incus2:
cpu flags count from API: 139
cached cpu flags count per member:
incus1.yaml: 141
incus2.yaml: cat: /var/cache/incus/resources/incus2.yaml: No such file or directory
0
incus3.yaml: 141
host incus3:
cpu flags count from API: 139
cached cpu flags count per member:
incus1.yaml: 141
incus2.yaml: 1
incus3.yaml: 141
I went through and removed the existing cache files and restarted the incus daemon on each of the nodes (not sure if the restart was actually necessary) and it seems to have updated them:
david@foobar ~ $ for host_name in incus{1..3}; do echo -e "\nhost ${host_name}:"; echo -n " cpu flags count from API: "; ssh ${host_name} "sudo incus query /1.0/resources | jq .cpu.sockets[0].cores[0].flags | wc -l"; echo " cached cpu flags count per member:"; for file_name in incus{1..3}.yaml; do echo -n " ${file_name}: "; ssh ${host_name} "sudo cat /var/cache/incus/resources/${file_name} | jq .cpu.sockets[0].cores[0].flags | wc -l"; done; done
host incus1:
cpu flags count from API: 139
cached cpu flags count per member:
incus1.yaml: cat: /var/cache/incus/resources/incus1.yaml: No such file or directory
0
incus2.yaml: 139
incus3.yaml: 139
host incus2:
cpu flags count from API: 139
cached cpu flags count per member:
incus1.yaml: 139
incus2.yaml: cat: /var/cache/incus/resources/incus2.yaml: No such file or directory
0
incus3.yaml: 139
host incus3:
cpu flags count from API: 139
cached cpu flags count per member:
incus1.yaml: 139
incus2.yaml: 139
incus3.yaml: cat: /var/cache/incus/resources/incus3.yaml: No such file or directory
0
On the YAML front, not sure why we’re doing JSON in a .yaml, but it’s still valid as YAML is a superset of JSON. We may want to actually switch to a YAML writer and reader though
