Is there a way to get a shell both in initramfs and on the fully-booted system in case the system gets into emergency mode to debug things? I’m stuck in initramfs for now and could do with a bit of information. I couldn’t find anything in the doc there (no obvious seed option). I’m using the latest image.
No, it’s specifically designed to prevent this as that would allow an attacker to interrupt boot, inject custom logic into the system and then resume the boot.
The initrd is completely read-only and all users are locked, the kernel boot parameters are part of the signed kernel image and cannot be modified. The booted system itself doesn’t have any user that are allowed for local login or any way to turn on a login prompt.
Was the system functioning correctly prior to that last update?
Any idea of what may have happened to get it into an unbootable state?
The only real recovery/debug option available is to pull the disk out of the system and plug it into another system, then using the recovery key you should be able to unlock the root partition and look at logs (assuming that you retrieved the recovery key).
I wasn’t expecting you to answer on a weekend. 
I understand what you say: that’s what drew me to incus OS to start with.
However I’m stuck on first boot (after successful secure registration followed by first boot install).
This is on a Dell Poweredge R650, clean install from IncusOS_202507032049. The system is not far enough in boot to dump emergency keys, AFAICT.
This is a clean install.
Okay, so just to confirm, you managed to boot from the install ISO, it did the secure boot enrollment and then you saw the install screen showing the data being copied and asking for you to remove the install media?
Then after you removed the install media and rebooted, you got this boot failure?
Precisely: custom ISO, SB enrollment, install screen, removing the virtual disk and reboot.
For completeness, adding incus keys was bumpy. Dell’s firmware filters out anything that does not end in .der (can’t see anything in the auth folder, which is problematic when having to import keys manually); wiping the platform key is a requirement to get into setup mode on that system; enrolling a new platform key wipes all existing DB keys (including the MS ones used for RAID card and network); restoring default DB keys deletes incus’ DB key… 
And when all is finally setup as intended, incus complained that SB wasn’t enabled (Dell requires that you boot in firmware to activate that manually).
But I got it installed and booting in the end, it installed, but failed after reboot, as shown above.
The way I see it, it might be worth it to give people an option, in the seed files, to explicitly add MS DB keys and ship that with Incus (bc. “append” mode does not seem to work everywhere).
Also bootctl used to be able to list and modify firmware options from the OS, I don’t know if it could speak directly with the BMC or if it was just IPMI magic but I believe there might be a way at least to detect that SB is not active and automatically reboot into firmware.
The only reason I can think of for failing at that boot stage would be an issue locating the installed system…
It’s actually interesting because we actually have a small test cluster of R650 running at a customer. And yeah, I definitely agree that the firmware setup on those, is only very moderately fun.
We’re currently busy switching all the signing keys to proper production keys for Incus OS. As part of that I plan on including the .der of all 3 keys (PK, KEK, DB) so that it’s easier to manually enroll them on such servers as the auto-enroll tends to be a bit problematic…
When I last installed on one of the R650, I found that it wasn’t too difficult to switch the Secure Boot policy to custom, then load the expected keys (assuming you have the .der) around, then cleanup DB a bit (basically only leaving the Incus OS key and the Microsoft key that’s used to sign the BOSS NVME driver).
Speaking of, I still need to have a chat with some DELL folks on why they thought it was a good idea to sign their BOSS NVME driver with the Microsoft key rather than with the same keys they use to sign the server firmware. That MS key is used to sign a LOT of stuff which isn’t exactly ideal when you’re trying to get a nice locked down environment.
would be an issue locating the installed system…
It does get into initramfs though, so the EFI partition is found. Then whether Incus does something smart with the disk ID provided in the seed or just rely on systemd’s DDI logic, it should work (DDI should only look for rootfs on same disk as EFI AFAIK). Hence my puzzlement.
I’ll wipe all the other disks however and report back: who knows what might still be on there.
We’re currently busy switching all the signing keys to proper production keys for Incus OS.
Exciting times, that’s why I wanted to get to the bottom of this now actually.
I found that it wasn’t too difficult to…
I was just suggesting that being able to inject extra SB DB certs through the seed process would make the process much more automatic, even on those servers, because “append mode” does not consistently work with automatic enrolling.
why they thought it was a good idea to sign their BOSS NVME driver with the Microsoft key
Well, I do wonder too, especially when they advertise this very use-case in their official semi-technical marketing:
The problem here is that DB key additions must be signed by the KEK, so we can’t easily let users inject arbitrary key. You can work around that while you’re in setup mode, but by the time we have an OS running, we’re no longer in setup mode.