Ah okay, I was trying to run proper Tumbleweed and Debian images and secureboot is enabled (which is what the default should be). This points more towards an issue with OVMF (I had issues with this before when packaging LXD ages ago) I guess. I can submit Incus to Factory for now until I figure out the secureboot issue, and then I’ll send it to SLE-Backports/Leap.
1 Like
The ethtool trick doesn’t work, and the guests are Linux so it seems quite strange that this (FreeBSD-specific?) problem would appear on Linux guests. The secureboot answer seems more likely to me. I suspect that the VM is failing to boot completely and the IPv6 allocation is being done during the creation of the bridge interface or during the VM setup, not by the VM itself.
Yes, I also think it has to do with OVMF.
I also port incus to clearlinux, and have a similar problem with secureboot.
Maybe @stgraber will have some hint?
That’s how I currently build EDK2 for the Zabbly packages, maybe the config and patches there will be useful.
1 Like
You can prove that one way or the other by looking at the console (serial and/or VGA) during startup.
I looked at the console when I recreated the issue, to the extent I could, and it was blank. The VM reboots or errors out or something happens too quickly for the console to show anything.
Since the problem doesn’t happen in the exact same VM when secureboot is disabled, but does occur when secureboot is re-enabled, I also think secureboot is where we have to start.
Unfortunately the console has no output (even with incus launch --console) and there is no output in the qemu log either. This is why it’s taken me a while to try to debug this.
@cyphar
I have managed to resolve this problem and fix secureboot VMs.
You have a folder of OVMF links which you make in your rpm build, which is given to incus as the environment variable INCUS_OVMF_PATH in incusd in the project. However, this has changed to INCUS_EDK2_PATH in recent incus versions.
We must change the line to:
export INCUS_EDK2_PATH=/usr/share/incus/ovmf
Without this variable set, incus was not using /usr/share/incus/qemu but rather using /usr/share/qemu, and choosing /usr/share/qemu/ovmf-x86_64-ms-4m-vars.bin for OVMF, which was causing the issues with the secureboot VMs as far as I can tell.
Just updating this environment variable will get the machines to not error anymore. Initially I thought that was all that was needed, but running mokutil --sb-state showed that although the machines booted and seemed fine, actually, secureboot was not working.
(I believe maybe until now with suse builds of lxd and incus, secureboot has never actually worked, but nobody ever checked 
)
So I looked into the way zabbly distributes the OVMF setup where secureboot does work, and after some testing finally determined which links were necessary to get it working.
Here are the needed links for Tumbleweed. They cover all of the .fd files available in zabbly’s distribution of incus, and seem to perform the same way in testing. Hopefully everything is the same for any other suse variants.
OVMF_CODE.4MB.fd -> /usr/share/qemu/ovmf-x86_64-4m-code.bin
OVMF_CODE.fd -> /usr/share/qemu/ovmf-x86_64-4m-code.bin
OVMF_VARS.4MB.fd -> /usr/share/qemu/ovmf-x86_64-4m-vars.bin
OVMF_VARS.4MB.ms.fd -> /usr/share/qemu/ovmf-x86_64-ms-4m-vars.bin
OVMF_VARS.fd -> /usr/share/qemu/ovmf-x86_64-4m-vars.bin
Without these links set up properly, the VM will boot and everything will seem fine, but secureboot will not be enabled when checked with mokutil --sb-state so it is necessary to modify the links.
When testing, please make sure to not use VMs which were set up before this change to test – they will give an error because the VM has the name of the OVMF file saved to it. Instead, please delete your old VMs and create new ones after this change, in order to test it.
As below, everything now seems fine:
microos:~ # incus launch images:debian/bookworm debvm-secureboot --vm -c security.secureboot=true
microos:~ # incus ls
+------------------+---------+----------------------+-------------------------------------------------+-----------------+-----------+
| NAME | STATE | IPV4 | IPV6 | TYPE | SNAPSHOTS |
+------------------+---------+----------------------+-------------------------------------------------+-----------------+-----------+
| debvm-secureboot | RUNNING | 10.6.122.65 (enp5s0) | fd42:5430:f748:f1e5:216:3eff:fec1:61c7 (enp5s0) | VIRTUAL-MACHINE | 0 |
+------------------+---------+----------------------+-------------------------------------------------+-----------------+-----------+
microos:~ # incus exec debvm-secureboot bash
root@debvm-secureboot:~# uname -r
6.1.0-27-amd64
root@debvm-secureboot:~#
microos:~ # incus exec debvm bash
root@debvm:~# mokutil --sb-state
SecureBoot enabled
Please see my obs branch here for exact changes.
I submitted a request to merge the changes.
Thanks @cmspam! The only remaining thing is figuring out a nice solution for the sysctl configuration. During the security audit, the reviewer suggested that we should either do the sysctl configuration as part of the daemon startup (ExecStartPre in a drop-in) or just drop the config entirely because it the sysctls are more for “production deployments” which admins can configure themselves if they find it necessary.
The main concern was that systemd that install both lxd and incus would now get both sets of sysctls applied, which can lead to strange results, and really the sysctls should only be applied once you have containers running on the machine instead of being global system settings.
@stgraber What do you think? Would you prefer the recommended sysctls should be dropped or made into something we apply before the daemon starts?
@cyphar @stgraber
Although the patch I submitted was a workaround, I traced the issue back to a bug in incus, which I submitted a PR for here.
Incus had some support for the ovmf names in openSUSE, but code and vars were swapped it turns out. So hopefully linking OVMF, and adding the environment variable will not be needed for the next version of incus.
I think it’s fine to skip them. Typically you only start running into problems with 100+ containers running systemd (keyring limits is the first thing you tend to hit). Not likely to be a problem for most regular users and those running at larger scale are likely to read our docs and apply the sysctls themselves.
1 Like
I’ve submitted incus to openSUSE. Sorry for the long delays everyone! 
3 Likes
Cheers!
I am very glad the barriers were overcome!
I look forward to setting up incus much more smoothly in microos soon!
@cyphar thanks, that’s great!
Once the package gets published, don’t forget to send us a PR for doc/installing.md so we have the instructions available for our users.
1 Like
Hi, great work.
Will the “incus-ui-canonical” also be added to openSUSE build/factory system later on ?
It’s been accepted into Tumbleweed and Leap.
2 Likes
Someone has made a package that just ships the built version from Zabbly, but if you want to include it into openSUSE itself we would want to have a proper source code tree that we can use (preferably without having to re-apply the patches that Zabbly does manually).
If the package was set up with a proper drop-in file (as I suggested in this submit request), you could install the experimental package and everything would work.