Incus VM on Raspberry PI4

Incus
1

Hi all,
following the hint provided by @stgraber closing #475 issue on github I have finally started a vm in Voidlinux x86 platform.
I can’t do the same on a Raspberry PI4.

Since Voidlinux doesn’t have the ovmf package for aarch64 platform I tried Fedora

To setup incus I followed instructions found at ‘Start your first virtual machine’ here, but i got the same QEMU error “Failed instance creation: Failed creating instance record: Instance type “virtual-machine” is not supported on this server: QEMU failed to run feature checks” even skipping the, to me unknown, cloud-init section.

Libvirtd.service is active (is that enough to run vm on Fedora??) and it looks like qemu is working.

@stgraber suggested to look into journalctl (no clues at all), and incusd log (no incusd log at all), so I ran out of options.

May anyone who started succesfully vm through incus on a Raspberry PI4 point me in the right direction?

My kind regards

P.S. Linux distribution doesn’t matter, I want only a working one

2

journalctl -u incus doesn’t show anything?

3

Simulating a VM detection failure on Fedora 39 here:

[root@f39 ~]# rm /dev/kvm 
[root@f39 ~]# systemctl restart incus
[root@f39 ~]# journalctl -u incus.
Mar 12 23:42:54 f39 systemd[1]: Starting incus.service - Incus - Daemon...
Mar 12 23:42:54 f39 incusd[1130]: time="2024-03-12T23:42:54Z" level=error msg="Unable to parse system idmap" err="No map found for user"
Mar 12 23:42:54 f39 incusd[1130]: time="2024-03-12T23:42:54Z" level=warning msg="AppArmor support has been disabled because of lack of kernel support"
Mar 12 23:42:54 f39 incusd[1130]: time="2024-03-12T23:42:54Z" level=warning msg=" - AppArmor support has been disabled, Disabled because of lack of kernel support"
Mar 12 23:42:54 f39 incusd[1130]: time="2024-03-12T23:42:54Z" level=warning msg="Instance type not operational" driver=qemu err="KVM support is missing (no /dev/kvm)" type=virtual-machine
Mar 12 23:42:55 f39 systemd[1]: Started incus.service - Incus - Daemon.
[root@f39 ~]#

This shows the expected error:

Mar 12 23:42:54 f39 incusd[1130]: time="2024-03-12T23:42:54Z" level=warning msg="Instance type not operational" driver=qemu err="KVM support is missing (no /dev/kvm)" type=virtual-machine
4

I use Raspberry Pi OS on my Raspberry Pis. It is basically just Debian 12.

Then you can use the Zabbly Debian packages of incus and zfs.

Incus runs perfectly on my 5, 4, and 3 models.

This is probably the easiest way of doing it.

5

My ignorance of systemd is total

Thanks to your suggestions i discovered where the problem. is

qemu-system-aarch64 device requires 67108864 bytes, block backend provides 1966080 bytes

I shortly googled for this error and it looks like the fix is over my capabilities

Sorry for the noise

6

I’ll try that way

Thanks

7

I don’t exactly understand what i did so this workaround could have unpredictable downsides, however…

cd /usr/share/edk2/aarch64
dd of="QEMU_EFI-pflash.raw" if="/dev/zero" bs=1M count=64
dd of="QEMU_EFI-pflash.raw" if="QEMU_EFI.fd" conv=notrunc
dd of="QEMU_VARS-pflash.raw" if="/dev/zero" bs=1M count=64
dd of="QEMU_VARS-pflash.raw" if="QEMU_VARS.fd" conv=notrunc
cd /usr/share/edk2/ovmf
dd of="OVMF_VARS.fd" if="/dev/zero" bs=1M count=64

allows the virtual machine to start

Next days i’ll do more tests

1 Like
8

Have you finally resolved the issue ?

I was migrated from LXD to Incus on RPI4 cluster run on top of Ubuntu 24.04 recently. I have the same issue, I’m not able to start VMs under Incus with the same error. I was able to run VMs on the LXD cluster before migration to the Incus.

Here list of qemu-system-* packages installed:

$ apt list --installed qemu-*
Listing... Done
qemu-block-extra/noble,now 1:8.2.2+ds-0ubuntu1 arm64 [installed,automatic]
qemu-efi-aarch64/noble,now 2024.02-2 all [installed,automatic]
qemu-efi-arm/noble,now 2024.02-2 all [installed,automatic]
qemu-system-arm/noble,now 1:8.2.2+ds-0ubuntu1 arm64 [installed,automatic]
qemu-system-common/noble,now 1:8.2.2+ds-0ubuntu1 arm64 [installed,automatic]
qemu-system-data/noble,now 1:8.2.2+ds-0ubuntu1 all [installed,automatic]
qemu-system-gui/noble,now 1:8.2.2+ds-0ubuntu1 arm64 [installed,automatic]
qemu-system-mips/noble,now 1:8.2.2+ds-0ubuntu1 arm64 [installed,automatic]
qemu-system-misc/noble,now 1:8.2.2+ds-0ubuntu1 arm64 [installed,automatic]
qemu-system-modules-opengl/noble,now 1:8.2.2+ds-0ubuntu1 arm64 [installed,automatic]
qemu-system-modules-spice/noble,now 1:8.2.2+ds-0ubuntu1 arm64 [installed,automatic]
qemu-system-ppc/noble,now 1:8.2.2+ds-0ubuntu1 arm64 [installed,automatic]
qemu-system-s390x/noble,now 1:8.2.2+ds-0ubuntu1 arm64 [installed,automatic]
qemu-system-sparc/noble,now 1:8.2.2+ds-0ubuntu1 arm64 [installed,automatic]
qemu-system-x86/noble,now 1:8.2.2+ds-0ubuntu1 arm64 [installed,automatic]
qemu-system/noble,now 1:8.2.2+ds-0ubuntu1 arm64 [installed]
qemu-utils/noble,now 1:8.2.2+ds-0ubuntu1 arm64 [installed,automatic]

Below is simple steps to reproduce:

incus launch images:alpine/edge test-vm  --vm
Launching test-vm
Error: Failed instance creation: Failed creating instance record: Instance type "virtual-machine" is not supported on this server: QEMU failed to run feature checks
9

How did you install Incus?

If from the Zabbly repository, make sure to apply updates as the initial upload of version 6.3 had an issue that would explain what you’re seeing.

10

Thank you Stéphane for quick reply!

As I wrote above, I migrated to Incus from LXD. The path is:

  • Upgrade from Ubuntu 22.04 + LXD 5.21 to Ubuntu 24.04 + LXD 5.21
  • Install Incus 6.0 package via apt install incus
  • Migration from LXD to Incus using lxd-to-incus
    Now the only 6.0.0-1 is available in Ubuntu repo.
11

Ah, okay, so the issue then may be the lack of OVMF firmwares on the system.

In the LXD case, those were bundled with the snap, but now that you’re using the .deb, you’ll need those to comes from a package and be available in a location that Incus looks at.

I’d start by installing the qemu-efi-aarch64 package if you don’t have it already.

12

Note that I don’t think that just installing the package will work. I think you’re going to have to copy some files or make some symlinks between the /usr/share/AAVMF/ entries and /usr/share/qemu.

Specifically, I think you’ll want:

  • /usr/share/AAVMF/AAVMF_CODE.fd => /usr/share/qemu/OVMF_CODE.fd
  • /usr/share/AAVMF/AAVMF_CODE.ms.fd => /usr/share/qemu/OVMF_CODE.ms.fd
  • /usr/share/AAVMF/AAVMF_VARS.fd => /usr/share/qemu/OVMF_VARS.fd
  • /usr/share/AAVMF/AAVMF_VARS.ms.fd => /usr/share/qemu/OVMF_VARS.ms.fd

Note that Incus 6.3 has logic to directly support /usr/share/AAVMF. That logic will be backported into 6.0.2 which may eventually find its way into the Ubuntu 24.04 repository as a bugfix release, but that’s mostly out of our control.

The alternative is to use the packages from GitHub - zabbly/incus: Incus package repository as those include an EDK2/UEFI build for both x86_64 and aarch64 so you don’t need to mess with symlinks to get that one working.

13

Thanks a lot Stéphane for the suggestions.
Quick check with links did not help. Links are:

$ ls -l /usr/share/qemu/OVMF*
lrwxrwxrwx 1 root root 15 Mar 11 07:25 /usr/share/qemu/OVMF.fd -> ../ovmf/OVMF.fd
lrwxrwxrwx 1 root root 30 Jul 14 21:09 /usr/share/qemu/OVMF_CODE.fd -> /usr/share/AAVMF/AAVMF_CODE.fd
lrwxrwxrwx 1 root root 33 Jul 14 21:10 /usr/share/qemu/OVMF_CODE.ms.fd -> /usr/share/AAVMF/AAVMF_CODE.ms.fd
lrwxrwxrwx 1 root root 30 Jul 14 21:10 /usr/share/qemu/OVMF_VARS.fd -> /usr/share/AAVMF/AAVMF_VARS.fd
lrwxrwxrwx 1 root root 33 Jul 14 21:10 /usr/share/qemu/OVMF_VARS.ms.fd -> /usr/share/AAVMF/AAVMF_VARS.ms.fd

However, incus reports on start:

Jul 14 21:16:43 cl-01 systemd[1]: Starting incus.service - Incus - Main daemon...
    Subject: A start job for unit incus.service has begun execution
    Defined-By: systemd
    Support: http://www.ubuntu.com/support

    A start job for unit incus.service has begun execution.

    The job identifier is 303.
Jul 14 21:16:44 cl-01 incusd[1277]: time="2024-07-14T21:16:44-07:00" level=warning msg=" - Couldn't find the CGroup blkio.weight, disk priority will be ignored"
Jul 14 21:16:45 cl-01 incusd[1277]: time="2024-07-14T21:16:45-07:00" level=error msg="Unable to run feature checks during QEMU initialization: QEMU premature exit: exit status 1 (qemu-system-aarch64: device requires 67108864 bytes, block backend provides 3653632 bytes)" 
Jul 14 21:16:45 cl-01 incusd[1277]: time="2024-07-14T21:16:45-07:00" level=warning msg="Instance type not operational" driver=qemu err="QEMU failed to run feature checks" type=virtual-machine
Jul 14 21:16:54 cl-01 incusd[1277]: time="2024-07-14T21:16:54-07:00" level=warning msg="Dqlite: attempt 1: server xxx.xxx.xxx.xxx:8443: no known leader"
Jul 14 21:17:35 cl-01 systemd[1]: Started incus.service - Incus - Main daemon.
    Subject: A start job for unit incus.service has finished successfully
    Defined-By: systemd
    Support: http://www.ubuntu.com/support
    A start job for unit incus.service has finished successfully.

I did not plan to use 3rd party repos, but I have to switch to zabbly to get better incus support.

14

Looks like that error got chopped off, any chance you can get the whole thing?

15

The message is:

Jul 14 21:16:45 cl-01 incusd[1277]: time="2024-07-14T21:16:45-07:00" level=error msg="Unable to run feature checks during QEMU initialization: QEMU premature exit: exit status 1 (qemu-system-aarch64: device requires 67108864 bytes, block backend provides 3653632 bytes)"

P.S.
I’m trying to upgrade incus from thezabbly now, but I have got unresolvable conflict. incus-base can not be installed probably due to conflict with lxd-to-incus' from the incus-tools` ubuntu package.

After trying apt install incus the dpkg failed to install incus-base package. Now the apt don’t allow me to do anything and require to fix installation using apt --fix-broken install which always fail with

sudo apt --fix-broken install
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
Correcting dependencies... Done
The following packages were automatically installed and are no longer required:
  incus-agent libcowsql0 liblxc-common libraft0 libsubid4 lxcfs uidmap
Use 'sudo apt autoremove' to remove them.
The following additional packages will be installed:
  incus-base
Suggested packages:
  zfsutils-linux | openzfs-zfsutils
The following NEW packages will be installed:
  incus-base
0 upgraded, 1 newly installed, 0 to remove and 5 not upgraded.
8 not fully installed or removed.
Need to get 0 B/69.4 MB of archives.
After this operation, 270 MB of additional disk space will be used.
Do you want to continue? [Y/n]
(Reading database ... 134949 files and directories currently installed.)
Preparing to unpack .../incus-base_1%3a6.3-202407130551-ubuntu24.04_arm64.deb ...
Unpacking incus-base (1:6.3-202407130551-ubuntu24.04) ...
dpkg: error processing archive /var/cache/apt/archives/incus-base_1%3a6.3-202407130551-ubuntu24.04_arm64.deb (--unpack):
 trying to overwrite '/usr/bin/lxd-to-incus', which is also in package incus-tools 6.0.0-1
Errors were encountered while processing:
 /var/cache/apt/archives/incus-base_1%3a6.3-202407130551-ubuntu24.04_arm64.deb
needrestart is being skipped since dpkg has failed
E: Sub-process /usr/bin/dpkg returned an error code (1)

Looks like I was able to resolve this conflict on one of nodes using next commands:

sudo dpkg --force-all --configure -a
sudo dpkg --purge --force-depends incus-tools
sudo apt --fix-broken install

In reality upgrade process ends with timeout as upgraded node is 6.3, but other nodes are 6.0. Whole cluster upgrade require a lot of manual intervention comparing to snap-based LXD.

16

Upgrading to 6.3 did not resolve Raspberry PI QEMU issue.
Incus reported on start-up (from /var/log/incus/incusd.log):

time="2024-07-14T22:44:11-07:00" level=error msg="Unable to run feature checks during QEMU initialization: Failed to query CPU model: The CPU type 'kvm64' is not a recognized ARM CPU type"
time="2024-07-14T22:44:11-07:00" level=warning msg="Instance type not operational" driver=qemu err="QEMU failed to run feature checks" type=virtual-machine
17

Hmm, that’s very odd as that’s something which was fixed in the Zabbly stable packages a few days ago…

Can you show dpkg -l | grep incus?

18 
ii  incus                                1:6.3-202407130551-ubuntu24.04          arm64        Incus - Container and virtualization daemon                                                                        
ii  incus-agent                          6.0.0-1                                 arm64        Incus guest agent                                                                                                 
ii  incus-base                           1:6.3-202407130551-ubuntu24.04          arm64        Incus - Container and virtualization daemon (container-only)                                                       
ii  incus-client                         1:6.3-202407130551-ubuntu24.04          arm64        Incus - Command line client

I need to clean-up incus-agent from the system as it’s not needed for incus 6.3.

If you see my first post you can see that qemu-system- are autoinstalled including qemu-system-x86. Does it matter?

19

Yeah, the incus-agent package can indeed go away, though it doesn’t really cause any issue other than take up some space on your disk :slight_smile:

The system QEMU also doesn’t matter when using the zabbly packages as it comes with its own QEMU build (you can look at the stuff it brings in in /opt/incus/bin).

Now as for the failure, I figured out what’s going on, but I’m now very confused as to how my own arm64 server has been running for the past few days given how obviously broken my previous fix was :wink:

Anyway, I just opened incusd/instance/qemu: Fix architecture check being backward by stgraber · Pull Request #1000 · lxc/incus · GitHub and will get it quickly pushed into the repo and then pushed to the Zabbly stable packages along with handling for incus-tools.

That should take a couple of hours to all be sorted out, I’ll post in here once that’s done.

20

I much appreciate for the quick resolution of this issue. I looked to this code last night and its logic was confused me a bit :).