I have a 3-node physical IncusOS cluster (comprising 3 x Dell Optiplex 5060 SFF), and wanted to add additional storage.
I evacuated one node, powered it off, and added the SATA drive. I restarted the node, and IncusOS booted up, but I wasn’t able to detect the drive using incus admin os storage show node03:.
I rebooted and enabled SATA-0 in the BIOS (previously disabled - this may or may not be relevant, but it was the only change that I made to the BIOS settings) and then rebooted to check that the BIOS could see the drive, which it could. I rebooted again:
IncusOS tried to enroll the secure boot keys - which failed due to a security violation
IncusOS tried to boot 202601100100, but failed due to not being in secure boot mode
IncusOS asked for the recovery key for disk root-x86-64
I went back to BIOS to check secure boot was enabled - it was (see below)
I rebooted and IncusOS tried again to enroll the secure boot keys again - failed
I tried to boot 202512250102 (the other option), but again failed due to not being in secure boot mode
IncusOS asked for the recover key for disk root-x86-64
I don’t have the recovery key - my bad, but I’ve been playing with the cluster and everything is deployed via Terraform (actually, OpenTofu) so I’m not worried, and I can rebuild this node and add it to the cluster.
I’ve checked the BIOS and it secure boot is enabled, secure boot is in audit mode (the alternative is deployed mode, which requires the Dell PK), and custom mode is enabled (I selected this when initially installing IncusOS).
I’ve also disabled SATA-0 in the BIOS, hoping that this might at least allow booting IncusOS, but this was not successful.
For the record, this drive was part of a ZFS storage pool which I destroyed and then zeroed the partition using cfdisk.
I’m OK to not proceed to rebuild this node right now, just in case there is any useful information that can be gleaned from it.
Your SecureBoot mode sounds problematic here, though not sure how changing SATA settings caused that to show up. It may actually be simply a consequence of moving to the new 2026 images which are signed by the 2026 key.
On DELL servers, you definitely want to make sure that you have:
Custom mode enabled
Enforcement should be “user”, you definitely don’t want “setup”, “audit”, or “deployed”
You need to make sure that our KEK is trusted and that our two DB keys are also trusted
The PK can be left to the factory one, that’s not a problem though you can also load ours as a replacement if you prefer
The nodes all set up initially without issue, with the secure boot keys being enrolled as expected during installation. I think I had to put secure boot into audit mode when installing - I can’t actually recall - but secure boot has been enabled from the start.
I’m going to re-initialise this node with an up-to-date IncusOS image, and try adding the SATA drive again. I’ll update this thread either way.
Sounds like (yet another) buggy UEFI implementation. The spec is pretty clear that when operating in Audit Mode, the SecureBoot EFI variable should be set to 0. IncusOS would then detect the lack of Secure Boot and display its usual warnings. If @accuser wasn’t seeing that, then SecureBoot was 1 when it shouldn’t have been.
It shouldn’t be too hard to add a bit of startup logic and refuse to boot if the AuditMode EFI variable exists and is non-zero.
It seems that Dell’s BIOS is detecting a hardware configuration change and interpreting this as potential tampering. Instead of refusing to boot entirely, the BIOS switches Secure Boot to Audit Mode as a compromise: it still performs signature verification but logs failures rather than blocking them, allowing you to boot and investigate. This doesn’t actually seem that secure to me.
I suspect that adding the SATA drive changed the storage controller enumeration, which triggered this.
I’ve just repeated the process with a second node:
Booted into BIOS and confirmed that secure boot was enabled (deployed mode, not audit mode). Enabled SATA-0 interface. Applied and restarted.
Booted into BIOS and noticed that SATA drive was detected, but secure boot was now in audit mode.
Booted and IncusOS tried to enrol secure boot keys, which failed.
A note on deployed vs. audit mode specifically for my case: these are the two secure boot options in the BIOS itself. There is a separate section where I enable custom keys, which I have used again to delete all keys before re-installing IncusOS.
I’m assuming that once the IncusOS keys have been enrolled, we’re running is user mode. To get there, I’ve enabled secure boot, enabled custom mode, and then deleted all keys (to get to setup mode), and then booted the IncusOS installation media and seen the keys enrol.
Ah, usually on DELL servers I’ve been doing the key enrollment by hand directly through the BIOS menu.
So going into the custom settings menu and then manually clear KEK, db and dbx, then load the new keys from the keys directory on the install media, leaving the manufacturer’s PK in place.
That way you get to make sure that all the settings are correct in the BIOS prior to ever booting the installer.
The only good Secure Boot mode for IncusOS is user. audit is problematic as it doesn’t do enforcement and could result in boot failure or encrypted storage access failure. deployed is also problematic because it prevents any further update to the Secure Boot keys by the OS.
IncusOS rotates its signing keys yearly, so we need a key update signed by the KEK to be able to enroll an new key into db. If running in deployed mode, this is blocked and effectively locks you onto whatever yearly key you deployed the server with, which will then prevent you from updating the server when we switch to new keys the following January (we just did 2025 → 2026 now).
All 3 nodes have been rebuilt and the cluster is back online.
The nodes are all in secure boot with custom mode enabled, which reports as user mode if I boot into Debian Live (I did this to check), so I’m sure that this is the nodes booting in secure boot user mode.
I did try the manual enrolment of keys, but the BIOS rejects them as being in the wrong format, but enabling secure boot (and rebooting) and then enabling custom mode and then deleting all keys (and rebooting) works.
Ah, that’s odd, we’ve installed quite a lot of DELL servers and they’ve always been happy to load the .der files. Though we’ve usually deployed on pretty recent servers.