Homeassistant VM won't boot after upgrade to 6.10 - Can't find root disk

Incus
1

Hi all,

I’ve been running a Home Assistant OS VM without issues for a while on Incus 6.9.

I just upgraded to 6.10 and the VM wouldn’t start anymore. In the VM console, it was looking for the root partition (console: “Waiting for root device PARTUUID=xxxx-xxxx…”), which is mapped from a normal block device on my default storage.

Looking at dmesg output, it seems like apparmor is denying access to ‘dev’ to qemu. Here are some of the relevant entries I see:

[Fri Feb 28 15:25:10 2025] audit: type=1400 audit(1740785110.869:42): apparmor="STATUS" operation="profile_remove" profile="unconfined" name="incus-homeassistant_</var/lib/incus>" pid=9619 comm="apparmor_parser"
[Fri Feb 28 15:37:09 2025] audit: type=1400 audit(1740785829.886:43): apparmor="STATUS" operation="profile_load" profile="unconfined" name="incus-homeassistant_</var/lib/incus>" pid=9901 comm="apparmor_parser"
[Fri Feb 28 15:37:10 2025] audit: type=1400 audit(1740785830.302:44): apparmor="DENIED" operation="open" profile="incus-homeassistant_</var/lib/incus>" name="/dev/bus/usb/" pid=9922 comm="qemu-system-x86" requested_mask="r" denied_mask="r" fsuid=997 ouid=0
[Fri Feb 28 15:37:10 2025] audit: type=1400 audit(1740785830.302:45): apparmor="DENIED" operation="open" profile="incus-homeassistant_</var/lib/incus>" name="/dev/" pid=9922 comm="qemu-system-x86" requested_mask="r" denied_mask="r" fsuid=997 ouid=0

I had not restarted the VM in a day or two, so I can’t 100% say that this only started after 6.10, but it seems likely.

I’ll try to set raw.apparmor on the VM settings to allow /dev access as suggested in other posts for now, hopefully that works.

Cheers!

2

I tried to recreate the VM from scratch (so importing a fresh image) to ensure it wasn’t something wrong with the disk mapping, but am getting the same error, with potentially apparmor denying disk access - unless I’m off base there.

I didn’t see anything in the 6.10 changelog that looked related, but perhaps the IOMMU change interacts with this in some way? It’s the only QEMU-related thing I saw.

3

I tried adding raw.apparmor: /dev/**/* rw, and variations to the VM config and while the apparmor denied message disappears, the VM still won’t boot, with the same error of not being able to find the root device. At this point I suspect it might be something else and am stumped.

4

Another possible cause could perhaps be some of the changes related to virtiofsd in 6.10?

Some more detail on steps to reproduce:

  1. Grab Home Assistant 14.2 (older versions have the same issue) from here
  2. Write appropriate short metadata.yaml file, tar it.
  3. incus image import metadata.tar haos_ova-14.2.qcow2 --alias haos
  4. incus linit haos homeassistant --vm -c security.secureboot=false
  5. incus start haos
  6. Console will show boot stuck at Waiting for root device PARTUUID=
5

Sorry to keep adding here as I seemingly find new info.

I tried to create a totally unrelated VM (NixOS from the iso installer) from scratch via:

incus init nixos --empty --vm \                                                                                                       
        -c security.secureboot=false \
        -c limits.cpu=4 \
        -c limits.memory=8GiB \
        -d root,size=64GiB

I uploaded the installer iso to my default storage and attached it to the VM with a boot priority of 10 as per the docs.

The installer boots, but it also can’t find any root disk. So something is going on with VM guests not seeing the default root disks attached to them on my end.

Here’s that VM’s config:

architecture: x86_64
config:
  limits.cpu: "4"
  limits.memory: 8GiB
  security.secureboot: "false"
  volatile.cloud-init.instance-id: 2840fcea-9a4d-4664-9a9a-bad428a78812
  volatile.eth0.host_name: mac6984e424
  volatile.eth0.hwaddr: 00:16:3e:8c:32:89
  volatile.eth0.last_state.created: "false"
  volatile.last_state.power: RUNNING
  volatile.uuid: 5fea993f-aaa0-45b6-bcaa-3bb2ab7eef9d
  volatile.uuid.generation: 5fea993f-aaa0-45b6-bcaa-3bb2ab7eef9d
  volatile.vsock_id: "4237013612"
devices:
  nixos-iso:
    boot.priority: "10"
    pool: default
    source: nixos-iso
    type: disk
  root:
    path: /
    pool: default
    size: 64GiB
    type: disk
ephemeral: false
profiles:
- default
- untagged
stateful: false
description: ""
6

Can you try setting io.bus=nvme to see if it’s an issue with virtio-scsi?

7

Hello, I just have the same problem after update from 6.6 to 6.10. I’m using ZFS volume for HAOS, inside it there are multiple partitions. I’ve checked that wanted partition exists, but not booting - VM is stuck on Waiting for root device PARTUUID=…

io.bus: nvme in disk session of yaml not helped.

I also want to say that’s bad that no older versions of incus are in Zabbly repository :frowning: (for Debian)

Installing incus LTS version won’t help, it doesn’t start with Error: Get “http://unix.socket/1.0”: EOF

8

Yes! People with the same issue as me! Spent hours on this yesterday, because it broke as the same time as I updated the kernel and ZFS to 2.3.0, so I wasn’t sure what went wrong.

But after rolling the kernel back it was still broken. There is definitely something wrong in the way the disk is exposed to the VM since the last incus version.

The home-assistant VM-layout looks like this

$ lsblk -o NAME,FSTYPE,SIZE,PARTLABEL,PARTUUID /dev/zvol/raid/incus/virtual-machines/haos.block
NAME    FSTYPE    SIZE PARTLABEL        PARTUUID
zd0                32G
├─zd0p1 vfat       32M hassos-boot      b3dd0952-733c-4c88-8cba-cab9b8b4377f
├─zd0p2 squashfs   24M hassos-kernel0   26700fc6-b0bc-4ccf-9837-ea1a4cba3e65
├─zd0p3 erofs     256M hassos-system0   8d3d53e3-6d49-4c38-8349-aff6859e82fd
├─zd0p4 squashfs   24M hassos-kernel1   fc02a4f0-5350-406f-93a2-56cbed636b5f
├─zd0p5 erofs     256M hassos-system1   a3ec664e-32ce-4665-95ea-7ae90ce9aa20
├─zd0p6             8M hassos-bootstate 33236519-7f32-4dff-8002-3390b62c309d
├─zd0p7 ext4       96M hassos-overlay   f1326040-5236-40eb-b683-aaa100a9afcf
└─zd0p8 ext4     31.3G hassos-data      a52a4597-fa3a-4851-aefd-2fbe9f849079

I’m looking at the changelog for 6.10, and I noticed the IOMMU change, maybe this is what is preventing HA from finding the partition?

9

It is indeed the IOMMU change :tada:

Editing the config to add raw.qemu.conf: '[device "qemu_iommu"]' to remove the IOMMU device fixes it and the VM start without issue.

3 Likes
10

I confirm that this is working with daily incus. Thank you very much, Max!

Stéphane, there should be at least one previous version in Zabbly repository to be able to downgrade in case of problems. Thanks.

11

Max’s workaround worked for me as well to get the VM starting again! Should this be the default for all blank VMs I create going forward?

12

i have the same issue. however this is not resolving my issue. I have multiple vm instances and all start normal except for homeassistant?

13

ps how can i mount the block device so i can retrieve my backup file inside the vm?

14

it worked

15

I can confirm it helped to boot my archlinux VM with the same issue not finding a root filesystem. Thank you, @halfa

16

it was discussed on this forum before, I asked the same question :slight_smile:

17

I can confirm this works. Thank you so much for sharing your findings!

18

I can confirm that I have experienced the same issue upon my upgrading from incus 6.9 to 6.10 and the proposed workaround/fix solving it - thank you @Max!

The affected installation is running on Raspberry CM4 with Ubuntu 24.04 and the latest Home Assistant OS 14.2. Before finding this post I’ve tried running earlier snapshots/versions of HassOS, but as you can imagine, got the same results.

19

Edit: I was using incus-migrate from apt so it was running 6.0. I manually downloaded the latest and it works now.

–
I’m trying to import Home Assistant for the first time and even adding the specified change, my VM still doesn’t boot. It’s just stuck on the Zabbly bootscreen with

BdsDxe: failed to load Boot0001 “UEFI QEMU QEMU HARDDISK " from PciRoot(0x0)/Pci(0x1,0x1)/Pci(0x0,0x0)/Scsi(0x0,0x1): Not Found”

I did an incus-migrate and imported as VM w/ secure boot off which I saw in another post. Any clue if this is related to this discussion or something else?

architecture: x86_64
config:
  raw.qemu.conf: '[device "qemu_iommu"]'
  security.secureboot: "false"
  volatile.cloud-init.instance-id: d8460e50-9383-4590-857b-31f44787b7fe
  volatile.eth0.host_name: tapb375f715
  volatile.eth0.hwaddr: 00:16:3e:30:e1:bf
  volatile.last_state.power: RUNNING
  volatile.last_state.ready: "false"
  volatile.uuid: c45f60e8-2329-4a8f-83ff-a17460a8423a
  volatile.uuid.generation: c45f60e8-2329-4a8f-83ff-a17460a8423a
  volatile.vsock_id: "602990596"
devices:
  root:
    path: /
    pool: default
    type: disk
ephemeral: false
profiles:
- default
- bridge-network
stateful: false
description: ""