LXD container live migration does not restore some processes

[abdou (Abderrahmane ABADA)]

I am using lxd move command to migrate a container. While this seems to be working and I can find that the container is running in the destination node, I noticed that in some cases, the background processes are not running. I can’t find a relationship between these cases for why this is happening.
The background processes are callback functions in a Python script running inside the container.
The issue is occurring in a ratio of approximately an error every 3 migrations. (No error in the logs, The process are killed silently).
Does anyone have an idea of the source of this issue?
The host nodes are running Ubuntu 18.04, I updated CRIU to the latest available version with no success.

Thank you in advance.

[gpatel-fr]

Hello

CRIU is not really working in a general way for LXD now:

Enable live migration?

Moving containers between nodes within a cluster can be done with renames now. CRIU integration for clustering hasn’t been touched though and isn’t a big priority for us given how unreliable CRIU generally is,.

Live migration with criu lxd latest with criu from sources

CRIU indeed does need a lot of work in order to be functional with common container setups. The majority of our active live migration users are working with workloads that are particularly designed to work with live migration. This typically involve unloading/unconfiguring anything that CRIU cannot handle (be that apparmor profiles, some network settings, …). Then ensuring that you don’t have any lxc exec sessions into the container as those also can’t be dumped and lastly reduce your process l…

[stgraber (Stéphane Graber)]

CRIU can be a bit weird at times.

If you can easily reproduce it, note down the PIDs and look at both the dump and restore logs to see if anything mentions that process and why it may have been skipped.

I’d also recommend you look for similar issues and if none match, file one at https://github.com/checkpoint-restore/criu

They’d obviously want you to test on the latest CRIU release (3.13).

[abdou (Abderrahmane ABADA)]

If anyone come across this post because of the background process being killed, I can confirm that the problem is not from CRIU.
In my case, the problem was because the killed process were having a connection with a remote server and if the communication fail (may be because of the down time in the move process) the connection is interrupted and the process are killed. I could resolve this problem by creating another process that check the connection after the migration has finished and if there is no connection then a request for a new connection will be established. It is working fine right now which confirm that the background process are not killed by a CRIU issue (the new monitoring process has never been killed).

[abdou (Abderrahmane ABADA)]

There was no logs regarding killed process, CRIU shows that all process are restored. As I said in the comment below, I do really believe that the issue is not with CRIU.