I think I found the culprit for the NVMe drives issue, even if I can’t explain why it doesn’t work.
As NVMe drives can report a different logical block address size, I wondered if changing the size would make a difference.
Here is the configuration I had before changing the LBA format:
$ sudo nvme id-ns -H /dev/nvme0n1
...
LBA Format 0 : Metadata Size: 0 bytes - Data Size: 512 bytes - Relative Performance: 0x2 Good
LBA Format 1 : Metadata Size: 0 bytes - Data Size: 4096 bytes - Relative Performance: 0x1 Better (in use)
Then, I changed the LBA format:
$ sudo nvme format --lbaf=0 /dev/nvme0n1
$ sudo nvme id-ns -H /dev/nvme0n1
...
LBA Format 0 : Metadata Size: 0 bytes - Data Size: 512 bytes - Relative Performance: 0x2 Good (in use)
LBA Format 1 : Metadata Size: 0 bytes - Data Size: 4096 bytes - Relative Performance: 0x1 Better
Finally I recreated the LVM pool and voilà:
$ incus storage create lvm lvm source=/dev/nvme0n1
$ incus launch images:ubuntu/24.04 v1 --vm --console -s lvm -c security.secureboot=false
Launching v1
To detach from the console, press: <ctrl>+a q
BdsDxe: loading Boot0001 "UEFI QEMU QEMU HARDDISK " from PciRoot(0x0)/Pci(0x1,0x1)/Pci(0x0,0x0)/Scsi(0x0,0x1)
BdsDxe: starting Boot0001 "UEFI QEMU QEMU HARDDISK " from PciRoot(0x0)/Pci(0x1,0x1)/Pci(0x0,0x0)/Scsi(0x0,0x1)
Ubuntu 24.04 LTS v1 ttyS0
v1 login:
EDIT: I reproduced the issue easily with an LVM storage pool on top of a custom loop device with a block size of 4k:
$ sudo truncate -s 20G /path/to/disk.img
$ sudo losetup -fP --show -b 4096 /path/to/disk.img
$ incus storage create lvm2 lvm source=/dev/loop0
$ incus launch images:ubuntu/24.04 v2 --vm --console -s lvm2 -c security.secureboot=false
Launching v2
To detach from the console, press: <ctrl>+a q
BdsDxe: failed to load Boot0001 "UEFI QEMU QEMU HARDDISK " from PciRoot(0x0)/Pci(0x1,0x1)/Pci(0x0,0x0)/Scsi(0x0,0x1): Not Found
>>Start PXE over IPv4.
Something is definitively wrong with LVM over a block device with a 4k block size. By default, the loop device created by Incus has a block size of 512 and therefore seems to work fine as with my SATA SSD (that has also a physical and logical block size of 512 from what I see)