I’m building a little appliance and want to be able to run some containers on it. So I selected Incus as my container manager and installed version 6.21
“Often” I can start containers and stop them. However, at some point “something happens”, and after that point all containers fail to start with something like:
$ incus start vm2
Error: Failed to run: /usr/bin/incusd forkstart vm2 /var/lib/incus/containers /run/incus/vm2/lxc.conf: exit status 1 (Failed to create lock for vm2
Error: Error initializing container for start: “allocating the container failed”)
Once one of these errors occurs then it will happen for every other container after that. And once you try and start the container it doesn’t seem possible to recover as it will then complain about the networking interfaces already being up, etc. If I take these down manually then it still fails with the same error.
However, the weird thing is if I simply run the “incusd forkstart blah” command line then it completes successfully and the container is started and appears to function ok for all intents and purposes?
Restarting incus doesn’t recover it. I’ve tried to poke with strace to see what resource it’s looking for and I can’t figure it out?? What is is waiting for. I tried a quick grep of the code to see if I could figure out where the error was coming from, but couldn’t figure it out
I’ve tried clearing every dir and resource I think incus could be trying to touch, so it’s got dirs in /run/incus and /var/lib/incus and some others from memory. Erasing those makes no difference
The kernel is an arm32 5.4.142. I’m trapped on a vendor kernel for now and the machine is an imx7 board. The instances are just small images and I’m using the file system as the backing store for everything.
The only thing which seems to recover the situation is a reboot, but that’s not ideal
Additionally, it seems as though I can reproduce this much more quickly if I have a simple web GUI interacting with the REST api of the incusd, not sure why, but something about this seems to put it in the blocked state almost instantly
Please help with some ideas on how to get more debugging out of this thing? I can’t see anything in the container boot logs, the system logs, possibly/probably I’m missing some way to get more info out of the command line tool itself?
Thanks