"Unable to create a stateful snapshot. CRIU isn't installed" even when criu.enabled = true

Hello,

I am getting an error as follows when I attempt to create a stateful snapshot:

lxc snapshot applications test --stateful
results in
Error: Unable to create a stateful snapshot. CRIU isn't installed

I am aware that criu.enable needs to be true

sudo snap get lxd criu shows

Key Value
criu.enable true

Of course, I rebooted after setting it, so it’s not a matter of the daemon needing to be restared.

snap list shows
lxd 3.18 12317 stable/…

Any ideas?

I had to install criu “apt install criu” then do the restart of lxd.

Error: Unable to create a stateful snapshot. CRIU isn't installed
root@p67:~# apt install criu
Reading package lists... Done
Building dependency tree
Reading state information... Done
criu is already the newest version (3.6-2).
The following packages were automatically installed and are no longer required:
  gyp libjs-async libjs-inherits libjs-node-uuid libuv1-dev node-abbrev
  node-ansi node-async node-balanced-match node-block-stream
  node-brace-expansion node-combined-stream node-concat-map node-cookie-jar
  node-delayed-stream node-forever-agent node-form-data node-fs.realpath
  node-fstream node-glob node-graceful-fs node-inflight node-inherits
  node-isexe node-json-stringify-safe node-mime node-minimatch node-mkdirp
  node-node-uuid node-nopt node-npmlog node-once node-osenv
  node-path-is-absolute node-qs node-request node-rimraf node-semver node-tar
  node-tunnel-agent node-which node-wrappy
Use 'sudo apt autoremove' to remove them.
0 upgraded, 0 newly installed, 0 to remove and 8 not upgraded.
root@p67:~# lxc snapshot ipsec1-aws-tf test --stateful
Error: Unable to create a stateful snapshot. CRIU isn't installed
root@p67:~# snap set lxd criu.enable=true
root@p67:~# lxc snapshot ipsec1-aws-tf test --stateful
Error: Unable to create a stateful snapshot. CRIU isn't installed
root@p67:~# systemctl reload snap.lxd.daemon
root@p67:~# lxc snapshot ipsec1-aws-tf test --stateful
Error: snapshot dump failed
(00.000147) Warn  (criu/log.c:203): The early log isn't empty
(03.663946) Warn  (criu/net.c:2910): Unable to get tun network namespace
(03.961593) Error (criu/files-ext.c:96): Can't dump file 32 of that type [600] (anon anon_inode:bpf-map)
(03.961675) Error (criu/cr-dump.c:1345): Dump files (pid: 19031) failed with -1
(03.971081) Error (criu/cr-dump.c:1743): Dumping FAILED.
root@p67:~#

Still get an error though:

root@p67:~# lxc snapshot ipsec1-aws-tf test --stateful
Error: snapshot dump failed
(00.000149) Warn  (criu/log.c:203): The early log isn't empty
(00.149759) Error (criu/files-ext.c:96): Can't dump file 32 of that type [600] (anon anon_inode:bpf-map)
(00.149839) Error (criu/cr-dump.c:1345): Dump files (pid: 19031) failed with -1
(00.156329) Error (criu/cr-dump.c:1743): Dumping FAILED

Your apt install was a no operation. LXD does not use the apt version, it use its own ‘vendored’ snap version.
The net effect of setting the criu.enable to true and restarting lxd (the logical way would be sudo snap restart lxd, since the change is at the snap level) is to set the path to load the snapified version of criu (that you can find in /snap/lxd/current/criu/). By looking at /proc//environ this can be seen quite easily.

sudo cat /proc/16688/environ
(…)
PATH=/snap/lxd/current/criu:/snap/(…)

Very probably the OP problem was coming from mixing ‘enable’ and ‘enabled’ and not restarting LXD after setting the correct value (enable, NOT enabled). Unfortunately Snap is allowing the user to set any value for a snap, even non existing ones.

sudo snap set lxd foo.baz=spam
sudo snap get lxd
Key Value
criu {…}
foo {…}

thanks for the info @gpatel-fr
I got the value “criu.enable” from this post: Is live migration with snap installed lxd 3.0.1 possible?

I tried setting both criu.enable and criu.enabled, restarted, and no dice.

...:~$ sudo snap set lxd criu.enabled=true
...:~$ sudo snap get lxd criu
Key Value
criu.enable true
criu.enabled true
...:~$ systemctl reload snap.lxd.daemon
...:~$ lxc snapshot dev-dwk --stateful
Error: Unable to create a stateful snapshot. CRIU isn't installed

regarding this suggestion: sudo cat /proc/16688/environ
to check the environment (forgive the stupid question), which process do I target? htop shows a number of processes referring to LXD, so wasn’t quite sure which one to look at.

root     27450  0.6  0.0   4500  1768 ?        Ss   17:13   0:00 /bin/sh /snap/lxd/12317/commands/daemon.start
root     27553  8.6  1.8 1217924 141716 ?      Sl   17:13   0:02  \_ lxd --logfile /var/snap/lxd/common/lxd/logs/lxd.log --group lxd

The second one (with the --logfile parameter)

So, there are a number of instances of that process. Taking one of them at random, I don’t see criu in the environ data. I rebooted and restarted containers before checking:

ARCH=x86_64-linux-gnuSNAP_INSTANCE_KEY=SNAP_COMMON=/var/snap/lxd/commonLXD_LXC_TEMPLATE_CONFIG=/snap/lxd/current/lxc/config/TEMPDIR=/tmpLD_LIBRARY_PATH=/snap/lxd/current/zfs-0.8/lib/:/var/lib/snapd/lib/gl:/var/lib/snapd/lib/gl32:/var/lib/snapd/void:/snap/lxd/12317/lib/x86_64-linux-gnu::/snap/lxd/12317/lib:/snap/lxd/12317/lib/x86_64-linux-gnu:/snap/lxd/current/lib:/snap/lxd/current/lib/x86_64-linux-gnuLISTEN_PID=2215LISTEN_FDS=1HOME=/tmp/SNAP_LIBRARY_PATH=/var/lib/snapd/lib/gl:/var/lib/snapd/lib/gl32:/var/lib/snapd/voidSNAP_USER_DATA=/root/snap/lxd/12317LXD_EXEC_PATH=/snap/lxd/current/bin/lxdLXD_DIR=/var/snap/lxd/common/lxd/LXD_SHIFTFS=falseSNAP_REVISION=12317TMPDIR=/tmpSNAP_CURRENT=/snap/lxd/currentXTABLES_LIBDIR=/snap/lxd/current/lib/xtables/JOURNAL_STREAM=9:40865SNAP_CONTEXT=TsLmyH3MJ6Pjb2vCFNHcyK9W30vnLXStTVwHq81w74svSNAP_VERSION=3.18SNAP_INSTANCE_NAME=lxdPATH=/snap/lxd/current/zfs-0.8/bin:/snap/lxd/12317/usr/sbin:/snap/lxd/12317/usr/bin:/snap/lxd/12317/sbin:/snap/lxd/12317/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/lxd/current/binINVOCATION_ID=c124f377676244268da58e01a13ea99fLISTEN_FDNAMES=unixXDG_RUNTIME_DIR=/run/user/0/snap.lxdSNAP_DATA=/var/snap/lxd/12317LXD_CLUSTER_UPDATE=/snap/lxd/current/commands/refreshLANG=en_US.UTF-8SNAP_USER_COMMON=/root/snap/lxd/commonSNAP_ARCH=amd64SNAP_COOKIE=TsLmyH3MJ6Pjb2vCFNHcyK9W30vnLXStTVwHq81w74svSNAP_REEXEC=SNAP_NAME=lxdLXD_LXC_HOOK=/snap/lxd/current/lxc/hooks/PWD=/home/snap/lxd/12317SNAP=/snap/lxd/12317

are you sure of that ? Does

ps fauxww | grep lxd | grep logfile

return more than one line ?

Nope, you’re right. I screwed it up the first time.

So the environ of that process is similar …
ARCH=x86_64-linux-gnuSNAP_INSTANCE_KEY=SNAP_COMMON=/var/snap/lxd/commonLXD_LXC_TEMPLATE_CONFIG=/snap/lxd/current/lxc/config/TEMPDIR=/tmpLD_LIBRARY_PATH=/var/lib/snapd/lib/gl:/var/lib/snapd/lib/gl32:/var/lib/snapd/void:/snap/lxd/12317/lib/x86_64-linux-gnu::/snap/lxd/12317/lib:/snap/lxd/12317/lib/x86_64-linux-gnu:/snap/lxd/current/lib:/snap/lxd/current/lib/x86_64-linux-gnuLISTEN_PID=478SNAP_LIBRARY_PATH=/var/lib/snapd/lib/gl:/var/lib/snapd/lib/gl32:/var/lib/snapd/voidHOME=/tmp/LISTEN_FDS=1SNAP_USER_DATA=/root/snap/lxd/12317LXD_EXEC_PATH=/snap/lxd/current/bin/lxdLXD_DIR=/var/snap/lxd/common/lxd/LXD_SHIFTFS=falseSNAP_REVISION=12317TMPDIR=/tmpSNAP_CURRENT=/snap/lxd/currentXTABLES_LIBDIR=/snap/lxd/current/lib/xtables/JOURNAL_STREAM=9:67703SNAP_CONTEXT=7pPmVFdSD3LnKR6PllcwKxFCnYcjD37KHYy3vyrdDlVHSNAP_VERSION=3.18SNAP_INSTANCE_NAME=lxdPATH=/snap/lxd/12317/usr/sbin:/snap/lxd/12317/usr/bin:/snap/lxd/12317/sbin:/snap/lxd/12317/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/lxd/current/binINVOCATION_ID=c4ebf70a895645a4bbfb38093a771c1cXDG_RUNTIME_DIR=/run/user/0/snap.lxdLISTEN_FDNAMES=unixSNAP_DATA=/var/snap/lxd/12317LXD_CLUSTER_UPDATE=/snap/lxd/current/commands/refreshLANG=C.UTF-8SNAP_ARCH=amd64SNAP_USER_COMMON=/root/snap/lxd/commonSNAP_COOKIE=7pPmVFdSD3LnKR6PllcwKxFCnYcjD37KHYy3vyrdDlVHSNAP_REEXEC=SNAP_NAME=lxdLXD_LXC_HOOK=/snap/lxd/current/lxc/hooks/PWD=/var/snap/lxd/12317SNAP=/snap/lxd/12317

Still don’t see any reference to criu :frowning_face:

yes, that’s the problem.

I have found the start script for lxd in /snap/lxd/current/commands/daemon.start, it contains this:

# Setup CRIU
if [ "${criu_enable:-"false"}" = "true" ]; then
    echo "==> Enabling CRIU"
    export PATH="${SNAP_CURRENT}/criu:${PATH}"
fi

so there should be something in a log somewhere, let’s see… yes that’s it

sudo journalctl -u snap.lxd.daemon

you should see something about CRIU just after some systemd messages in your log. If not that’s because criu.enable is not equal to ‘true’ and if you have set it up correctly it could be snap’s fault if the lxd start script don’t see it.

So looking in the log, there is nothing about CRIU mentioned. It would seem that for some reason the daemon startup script is failing to see that criu is enabled.

I tried something as a total hack to try to confirm this theory…

I copied the contents /snap/lxd/current/commands to ~/tmp and then did this:
sudo mount --bind -o nodev,ro /home/$USER/tmp /snap/lxd/current/commands/

Then I modified the daemon.start in the ~/tmp directory …

# Setup CRIU
# if [ "${criu_enable:-"false"}" = "true" ]; then
    echo "==> Enabling CRIU"
    export PATH="${SNAP_CURRENT}/criu:${PATH}"
# fi

… and I restarted the daemon. Sure enough, the log now shows CRIU being enabled. Next, I tried to create a stateful snapshot. This time, the error about CRIU not bieng installed was gone. There was in fact another error that seems unrelated (Error: mkdir /var/snap/lxd/common/lxd/containers/dev-dwk: file exists) but I’m not worried about that since obviously this is a total hack for testing only.

The main thing I wanted to prove is that for some reason, the LXD startup is not seeing the configuration key.

Any thoughts about where to go from here?

Seems convincing, that’s the main point.

Not immediately. Anyway I’m going to sleep soon.

If in 24 hours you have no further help (from me or someone else) in your place I’d think of backup your containers / uninstall lxd and snap itself and install again, restore your containers if necessary and try again. There seems to be something wrong in the snap / lxd interaction, maybe something is broken.

1 Like

It will be a while before I’ll be able to test out the approach. Marking the thread as resolved for now. I’ll come back later with concrete results.