Old lxd/incus Vms wont boot!

Incus
,
1

Hi all, i am using incus vms and containers for a long time now. i immigrate to incus almost a year a go and i have a bunch of old vms and containers.
since yestersday iunderestood all my old vms wont boot and the reason is under /boot directory there is no kernel exist! i tried to recreat a kernel, didnt help!
can someone please help?
why only my old vms effected?

i am using archlinux and latest version of incus
if you need more info please let me know

2

Welcome!

Can you check whether you are affected by this, Incus 6.5 -- Incus VM not starting in Cluster
Although the title says that’s related to cluster environments, the issue was present also on standalone installations.

  1. Run incus version to verify whether you are really running version 6.5 (affected version).
  2. When Incus 6.5 was first released, there was this bug. It was identified quickly, fixed and new packages were created for Debian and Ubuntu. Perhaps for Arch you need to get refreshed packages? Is there a way to check whether Arch has 6.5 packages that have been updated with the fix?

You mention though that it is an issue with old VMs only, while newer VMs are OK.
When you upgrade to newer versions of Incus, the Qemu virtualization and emulator software may also get updated. Can you verify your version?

$ /opt/incus/bin/qemu-system-x86_64 --version
QEMU emulator version 9.0.2 (v9.0.2-dirty)
Copyright (c) 2003-2024 Fabrice Bellard and the QEMU Project developers
$

There is a chance that there is some kind regression of Qemu 9.0.2 with other images.

On your point, you mention that /boot is empty in those VMs. Can you tell us which images you are running, and also what are the contents of /etc/fstab in the affected VMs. Obviously, we are looking whether those images require the mounting of /boot which somehow did not happen.

3

Thanks for your answer,
i checked every thing on one my servers which i had 2 vms affected :

qemu-system-x86_64 --version
QEMU emulator version 9.1.0
Copyright (c) 2003-2024 Fabrice Bellard and the QEMU Project developers

also it seems for archlinux version 6.5 is not out yet :
❯

 pacman -Q | grep incus
incus 6.4-1
❯ incus version
Client version: 6.4
Server version: unreachable

on the effected vms i missed initramfs-linux.img and vmlinuz-linux image and kernel :

sudo ls /boot
efi  grub initramfs-linux.img  vmlinuz-linux

under /etc/fstab :

sudo cat /etc/fstab 
LABEL=rootfs  /         ext4  defaults  0 0
LABEL=UEFI    /boot/efi vfat  defaults  0 0

Although it appears I’m not using an affected version of Incus, what other factors could cause my kernel to be deleted?

4

It does not appear that the missing files in /boot/ are related to that directory being mounted from some other location.

Can you tell me what was your base image for those old VMs?
It should be something like ubuntu:20.04 or ubuntu:focal.
Also, when did you install them? (like 2 years ago, etc).
By doing so, it would be possible to pull those old versions into a VM that runs LXD, then migrate to Incus, and check if they fail.

Also, can you verify that you are running these on ArchLinux? (the host)

5

All of my main servers (hosts) and VMs are running Arch Linux. I’m not sure what the original image was, as they were created nearly five years ago and migrated to Incus last year.
maybe this : images:archlinux/cloud/amd64

Fortunately, I was able to restore the VMs from snapshots, reinstall Arch Linux, and recreate the kernel. However, I still don’t understand what could have caused the kernel to be deleted. I’m concerned this might happen again, as I have important certificates and configurations on those systems.