UEFI boot order does not persist after reboot

mkgvt · September 22, 2026, 7:55pm

I am trying to understand how to control the boot order of an Incus VM. The eventual goal is to PXE/HTTP boot without using the hard drive.

Set up:

$ incus launch images:alpine/3.24 bootorder --vm -c security.secureboot=false
$ incus shell bootorder
# apk add efibootmgr  # install

Initial boot order (with some details elided for conciseness):

# efibootmgr
BootCurrent: 0002
Timeout: 3 seconds
BootOrder: 0002,0003,0004,0005,0006,0007,0008,0009,000A,0000,0001
Boot0000* BootManagerMenuApp ...
Boot0001* EFI Firmware Setup ...
Boot0002* UEFI QEMU QEMU HARDDISK ...
Boot0003* UEFI PXEv4  ...
Boot0004* UEFI PXEv6  ...
Boot0005* UEFI HTTPv4  ...
Boot0006* UEFI HTTPv6  ...
Boot0007* UEFI PXEv4  ...
Boot0008* UEFI PXEv6  ...
Boot0009* UEFI HTTPv4  ...
Boot000A* UEFI HTTPv6  ...

Change the boot order and shut the VM down:

# efibootmgr -o 0003
BootCurrent: 0002
Timeout: 3 seconds
BootOrder: 0003
Boot0000* BootManagerMenuApp ...
Boot0001* EFI Firmware Setup ...
Boot0002* UEFI QEMU QEMU HARDDISK ...
Boot0003* UEFI PXEv4  ...
Boot0004* UEFI PXEv6  ...
Boot0005* UEFI HTTPv4  ...
Boot0006* UEFI HTTPv6  ...
Boot0007* UEFI PXEv4  ...
Boot0008* UEFI PXEv6  ...
Boot0009* UEFI HTTPv4  ...
Boot000A* UEFI HTTPv6  ...
# halt

Note: Alpine doesn’t have shutdown, only halt which doesn’t close the Incus shell like shutdown -h now does in Debian. The VM has to be explicitly stopped.

$ incus stop -f bootorder

Restart. I expect it to only attempt to PXEv4 boot (and fail as PXE is not set up). Instead, it boots from the disk. It never attempts to boot over PXEv4.

$ incus start bootorder
To detach from the console, press: <ctrl>+a q
BdsDxe: loading Boot0002 "UEFI QEMU QEMU HARDDISK " from PciRoot(0x0)/Pci(0x1,0x1)/Pci(0x0,0x0)/Scsi(0x0,0x1)
BdsDxe: starting Boot0002 "UEFI QEMU QEMU HARDDISK " from PciRoot(0x0)/Pci(0x1,0x1)/Pci(0x0,0x0)/Scsi(0x0,0x1)
  Booting `Alpine Linux v3.24'

Loading Linux virt ...
Loading initial ramdisk ...
... elided ...
Welcome to Alpine Linux 3.24
Kernel 6.18.52-0-virt on x86_64 (/dev/console)

bootorder login:

Check the boot order. It has been reset.

$ incus exec bootorder -- efibootmgr
BootCurrent: 0002
Timeout: 3 seconds
BootOrder: 0002,0003,0004,0005,0006,0007,0008,0009,000A,0000,0001
Boot0000* BootManagerMenuApp ...
Boot0001* EFI Firmware Setup ...
Boot0002* UEFI QEMU QEMU HARDDISK ...
Boot0003* UEFI PXEv4  ...
Boot0004* UEFI PXEv6  ...
Boot0005* UEFI HTTPv4  ...
Boot0006* UEFI HTTPv6  ...
Boot0007* UEFI PXEv4  ...
Boot0008* UEFI PXEv6  ...
Boot0009* UEFI HTTPv4  ...
Boot000A* UEFI HTTPv6  ...

It appears that Incus generates the UEFI variables each time it boots. Is this correct? How does one control the boot order?

Notes:

This VM has two interfaces on different Incus networks which is why there are two sets of PXE and HTTP entries.
The boot order is reset with images:debian/13 as well so this is not an Alpine thing.
Eventually, I would like to boot via HTTP only. I discovered that PXEv4/PXEv6 UEFI entries can be removed via
```
raw.qemu: |
  -fw_cfg name=opt/org.tianocore/IPv4PXESupport,string=no
  -fw_cfg name=opt/org.tianocore/IPv6PXESupport,string=no
```
but I don’t know how to remove any of the other entries. It might work if I could remove the hard disk entry.
No hard disk at all would be nice but Incus requires one even though it won’t be used.

stgraber · September 22, 2026, 10:22pm

Back when we were still running a bunch of network booted VMs in our lab, I had no problem having the boot order persist.

What Incus version on what distribution is this?

mkgvt · September 22, 2026, 10:35pm

IncusOS:

$ incus admin os info 
...
OS: IncusOS 202608201218
...
Applications:
  incus (7.4 [202609150130])
...

Incus client on NixOS 26.05:

$ incus --version
7.4

bensmrs · September 23, 2026, 1:19am

poweroff should do the job on Alpine.

Not quite, it’s actually EDK2 who generates BootOrder. Incus handles boot order through the boot.priority configuration keys for disk and nic devices. Those make QEMU inject custom entries in its bootorder fw_cfg key (which you can see with cat /sys/firmware/qemu_fw_cfg/by_name/bootorder/raw; on a brand new VM with a single NIC, it looks a bit like this:

/pci@ffffffffffffffff/pci-bridge@1,1/scsi@0/channel@0/disk@0,1
/pci@ffffffffffffffff/pci-bridge@1,4/ethernet@0/ethernet-phy@0
HALT

EDK2 then looks at these and matches them against the boot entries stored in your NVRAM. /pci@ffffffffffffffff/pci-bridge@1,1/scsi@0/channel@0/disk@0,1 should match against Boot0002, and /pci@ffffffffffffffff/pci-bridge@1,4/ethernet@0/ethernet-phy@0 should match against Boot0003, Boot0004, Boot0005 and Boot0006. On your system, it looks like you have a second NIC, which would correspond to Boot0007, Boot0008, Boot0009 and Boot000A. EDK2 then appends miscellaneous EFI applications, which correspond to Boot0000 and Boot0001.

Whatever you do to fight this, EDK2 is here to remind you who’s the boss

mkgvt:

Eventually, I would like to boot via HTTP only. I discovered that PXEv4/PXEv6 UEFI entries can be removed via
raw.qemu: |
  -fw_cfg name=opt/org.tianocore/IPv4PXESupport,string=no
  -fw_cfg name=opt/org.tianocore/IPv6PXESupport,string=no

Eh, that’s a sad hack

I think I do. @stgraber should we make the boot priority actually optional and not automatically computed, such that one can pass a special boot.priority: disable or none value to prevent this situation? I can send a patch, it should be pretty trivial.
I’m not even sure it makes sense to auto-generate the priorities by default, and think that disable/none should be the new default instead of 0 (except for the root device, which has another default anyway), but that may break existing installations; WDYT?

As for disabling only HTTP or PXE, I’ll have to think about new configuration keys.

I kinda agree with this one; I’d love Incus to support VMs without a root disk.

stgraber · September 23, 2026, 1:26am

Yeah, a special boot.priority: disable would be nice to have

mkgvt · September 23, 2026, 1:06pm

Thanks Ben. I’d forgotten about that one.

So I am learning.

bensmrs:

mkgvt:
Eventually, I would like to boot via HTTP only. I discovered that PXEv4/PXEv6 UEFI entries can be removed via
raw.qemu: |
  -fw_cfg name=opt/org.tianocore/IPv4PXESupport,string=no
  -fw_cfg name=opt/org.tianocore/IPv6PXESupport,string=no
Eh, that’s a sad hack

No argument here. I look forward to the boot.priority: disabled or none you suggested. It will solve the problem elegantly. In the mean time, I’ll change their priority so they get out of the way without resorting to gross hacks.

One of my use cases may be a bit different. Not only do I use Incus to implement services, but I also use it to prototype systems before I “go into the data center” so I have a handle up front on the problems I will face. All but one of the servers in the current project are to be net booted and won’t have disks. So the requirement that Incus VMs have a root disk is a (minor) deviation from how the physical systems will be implemented. It would be nice if VMs didn’t need a root disk. Not a big deal but desirable from an elegance standpoint.