Unable to stop a container that was moved from lxd to incus

Incus
1

I have a container running in Incus, but it’s state is totally unusable. Also, I cannot stop it, not even with the --force flag.

First let me explain how this came to be. It’s a fairly old Ubuntu 16.04 container that I had running in LXD 5.0. I’ve setup a new server with Incus 6.0. I am able to move containers from the old LXD host onto the new Incus server.

After the move (which succeeded without error) I started the container, but then I noticed it wasn’t working at all. It is now stuck in state RUNNING. Even after restarting the Incus server that container is again in RUNNING state. So far, I have not been able to stop it.

I can incus shell into the container. There are just three processes running. /sbin/init, su -l and -su.

Can this be caused by moving the container from LXD to Incus?

2

First thing I want to achieve: how can I really stop this container?

I was able to make a copy with incus copy. After that I was able to forcefully delete the container. For now I’ve copied the container back to the LXD server.

Now, what do I have to do to be able to move the container from LXD to Incus? Is there a conversion step to be taken? (Note. This is a Ubuntu 16.04 container.)

3

So far, moving containers with Ubuntu 18 and up goes without problems. There is definitely something wrong with migrating containers with Ubuntu 16 and older.

4

Is the container itself Ubuntu 16.04?
If so, you’re probably hitting some cgroup1/cgroup2 issue when moving to a more recent server.

That said, I don’t know why incus stop -f wouldn’t work.

Those containers certainly would fail to fully start and you’d see a very limited process list as you described, you’d also see some systemd errors in incus info --show-log NAME, but killing the container with incus stop -f should still work.

If it doesn’t, it may be useful to look at the ps fauxww output following the failed incus stop -f to see if anything weird is going on with the processes running that container.

5

The LXD host runs Ubuntu 20.04
The new Incus server runs Ubuntu 22.04
The problematic conainers run Ubuntu 16.04

Tomorrow I’ll add more logging of what I see running inside the container.

6

If it doesn’t, it may be useful to look at the ps fauxww output following the failed incus stop -f to see if anything weird is going on with the processes running that container.

That’s ps fauxwww on the Incus server, right?

7

Right, so Ubuntu 20.04 is cgroup1, Ubuntu 22.04 is cgroup2.
Your container only works with cgroup1 so it will not be able to function on the new host.

You’d need to force the Ubuntu 22.04 host back into cgroup1 mode to get that container to behave.

8

Yep

9

Ah, thanks for confirming.

Do you have a suggestion how I can best migrate Ubuntu 16 containers to the new Incus (cgroup2) server?

10

You can’t. Ubuntu 16.04 and similar older OS like CentOS 7 will simply not boot on a cgroup2 box, systemd just crashes immediately on startup.

Your only real option is to force the system back into cgroup1 mode.

11

But note that you’ll expose yourself to the opposite problem then as newer versions of systemd (possibly 24.04, not sure) no longer support cgroup1 and will refuse to boot.

So you basically end up having to run some systems on cgroup1 for your old containers and some systems on cgroup2 for the newer ones…

12

OK. Again, thanks for the explanation.

13

Or create a VM with cgroup1, and move the containers inside that VM.