Hello,
I am getting an error as follows when I attempt to create a stateful snapshot:
lxc snapshot applications test --stateful
results in
Error: Unable to create a stateful snapshot. CRIU isn't installed
I am aware that criu.enable needs to be true
sudo snap get lxd criu shows
Key Value
criu.enable true
Of course, I rebooted after setting it, so it’s not a matter of the daemon needing to be restared.
snap list shows
lxd 3.18 12317 stable/…
Any ideas?
I had to install criu “apt install criu” then do the restart of lxd.
Error: Unable to create a stateful snapshot. CRIU isn't installed
root@p67:~# apt install criu
Reading package lists... Done
Building dependency tree
Reading state information... Done
criu is already the newest version (3.6-2).
The following packages were automatically installed and are no longer required:
gyp libjs-async libjs-inherits libjs-node-uuid libuv1-dev node-abbrev
node-ansi node-async node-balanced-match node-block-stream
node-brace-expansion node-combined-stream node-concat-map node-cookie-jar
node-delayed-stream node-forever-agent node-form-data node-fs.realpath
node-fstream node-glob node-graceful-fs node-inflight node-inherits
node-isexe node-json-stringify-safe node-mime node-minimatch node-mkdirp
node-node-uuid node-nopt node-npmlog node-once node-osenv
node-path-is-absolute node-qs node-request node-rimraf node-semver node-tar
node-tunnel-agent node-which node-wrappy
Use 'sudo apt autoremove' to remove them.
0 upgraded, 0 newly installed, 0 to remove and 8 not upgraded.
root@p67:~# lxc snapshot ipsec1-aws-tf test --stateful
Error: Unable to create a stateful snapshot. CRIU isn't installed
root@p67:~# snap set lxd criu.enable=true
root@p67:~# lxc snapshot ipsec1-aws-tf test --stateful
Error: Unable to create a stateful snapshot. CRIU isn't installed
root@p67:~# systemctl reload snap.lxd.daemon
root@p67:~# lxc snapshot ipsec1-aws-tf test --stateful
Error: snapshot dump failed
(00.000147) Warn (criu/log.c:203): The early log isn't empty
(03.663946) Warn (criu/net.c:2910): Unable to get tun network namespace
(03.961593) Error (criu/files-ext.c:96): Can't dump file 32 of that type [600] (anon anon_inode:bpf-map)
(03.961675) Error (criu/cr-dump.c:1345): Dump files (pid: 19031) failed with -1
(03.971081) Error (criu/cr-dump.c:1743): Dumping FAILED.
root@p67:~#
Still get an error though:
root@p67:~# lxc snapshot ipsec1-aws-tf test --stateful
Error: snapshot dump failed
(00.000149) Warn (criu/log.c:203): The early log isn't empty
(00.149759) Error (criu/files-ext.c:96): Can't dump file 32 of that type [600] (anon anon_inode:bpf-map)
(00.149839) Error (criu/cr-dump.c:1345): Dump files (pid: 19031) failed with -1
(00.156329) Error (criu/cr-dump.c:1743): Dumping FAILED
Your apt install was a no operation. LXD does not use the apt version, it use its own ‘vendored’ snap version.
The net effect of setting the criu.enable to true and restarting lxd (the logical way would be sudo snap restart lxd, since the change is at the snap level) is to set the path to load the snapified version of criu (that you can find in /snap/lxd/current/criu/). By looking at /proc//environ this can be seen quite easily.
sudo cat /proc/16688/environ
(…)
PATH=/snap/lxd/current/criu:/snap/(…)
Very probably the OP problem was coming from mixing ‘enable’ and ‘enabled’ and not restarting LXD after setting the correct value (enable, NOT enabled). Unfortunately Snap is allowing the user to set any value for a snap, even non existing ones.
sudo snap set lxd foo.baz=spam
sudo snap get lxd
Key Value
criu {…}
foo {…}
thanks for the info @gpatel-fr
I got the value “criu.enable” from this post: Is live migration with snap installed lxd 3.0.1 possible?
I tried setting both criu.enable and criu.enabled, restarted, and no dice.
...:~$ sudo snap set lxd criu.enabled=true
...:~$ sudo snap get lxd criu
Key Value
criu.enable true
criu.enabled true
...:~$ systemctl reload snap.lxd.daemon
...:~$ lxc snapshot dev-dwk --stateful
Error: Unable to create a stateful snapshot. CRIU isn't installed
regarding this suggestion: sudo cat /proc/16688/environ
to check the environment (forgive the stupid question), which process do I target? htop shows a number of processes referring to LXD, so wasn’t quite sure which one to look at.
root 27450 0.6 0.0 4500 1768 ? Ss 17:13 0:00 /bin/sh /snap/lxd/12317/commands/daemon.start
root 27553 8.6 1.8 1217924 141716 ? Sl 17:13 0:02 \_ lxd --logfile /var/snap/lxd/common/lxd/logs/lxd.log --group lxd
The second one (with the --logfile parameter)
So, there are a number of instances of that process. Taking one of them at random, I don’t see criu in the environ data. I rebooted and restarted containers before checking:
ARCH=x86_64-linux-gnuSNAP_INSTANCE_KEY=SNAP_COMMON=/var/snap/lxd/commonLXD_LXC_TEMPLATE_CONFIG=/snap/lxd/current/lxc/config/TEMPDIR=/tmpLD_LIBRARY_PATH=/snap/lxd/current/zfs-0.8/lib/:/var/lib/snapd/lib/gl:/var/lib/snapd/lib/gl32:/var/lib/snapd/void:/snap/lxd/12317/lib/x86_64-linux-gnu::/snap/lxd/12317/lib:/snap/lxd/12317/lib/x86_64-linux-gnu:/snap/lxd/current/lib:/snap/lxd/current/lib/x86_64-linux-gnuLISTEN_PID=2215LISTEN_FDS=1HOME=/tmp/SNAP_LIBRARY_PATH=/var/lib/snapd/lib/gl:/var/lib/snapd/lib/gl32:/var/lib/snapd/voidSNAP_USER_DATA=/root/snap/lxd/12317LXD_EXEC_PATH=/snap/lxd/current/bin/lxdLXD_DIR=/var/snap/lxd/common/lxd/LXD_SHIFTFS=falseSNAP_REVISION=12317TMPDIR=/tmpSNAP_CURRENT=/snap/lxd/currentXTABLES_LIBDIR=/snap/lxd/current/lib/xtables/JOURNAL_STREAM=9:40865SNAP_CONTEXT=TsLmyH3MJ6Pjb2vCFNHcyK9W30vnLXStTVwHq81w74svSNAP_VERSION=3.18SNAP_INSTANCE_NAME=lxdPATH=/snap/lxd/current/zfs-0.8/bin:/snap/lxd/12317/usr/sbin:/snap/lxd/12317/usr/bin:/snap/lxd/12317/sbin:/snap/lxd/12317/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/lxd/current/binINVOCATION_ID=c124f377676244268da58e01a13ea99fLISTEN_FDNAMES=unixXDG_RUNTIME_DIR=/run/user/0/snap.lxdSNAP_DATA=/var/snap/lxd/12317LXD_CLUSTER_UPDATE=/snap/lxd/current/commands/refreshLANG=en_US.UTF-8SNAP_USER_COMMON=/root/snap/lxd/commonSNAP_ARCH=amd64SNAP_COOKIE=TsLmyH3MJ6Pjb2vCFNHcyK9W30vnLXStTVwHq81w74svSNAP_REEXEC=SNAP_NAME=lxdLXD_LXC_HOOK=/snap/lxd/current/lxc/hooks/PWD=/home/snap/lxd/12317SNAP=/snap/lxd/12317
are you sure of that ? Does
ps fauxww | grep lxd | grep logfile
return more than one line ?
Nope, you’re right. I screwed it up the first time.
So the environ of that process is similar …
ARCH=x86_64-linux-gnuSNAP_INSTANCE_KEY=SNAP_COMMON=/var/snap/lxd/commonLXD_LXC_TEMPLATE_CONFIG=/snap/lxd/current/lxc/config/TEMPDIR=/tmpLD_LIBRARY_PATH=/var/lib/snapd/lib/gl:/var/lib/snapd/lib/gl32:/var/lib/snapd/void:/snap/lxd/12317/lib/x86_64-linux-gnu::/snap/lxd/12317/lib:/snap/lxd/12317/lib/x86_64-linux-gnu:/snap/lxd/current/lib:/snap/lxd/current/lib/x86_64-linux-gnuLISTEN_PID=478SNAP_LIBRARY_PATH=/var/lib/snapd/lib/gl:/var/lib/snapd/lib/gl32:/var/lib/snapd/voidHOME=/tmp/LISTEN_FDS=1SNAP_USER_DATA=/root/snap/lxd/12317LXD_EXEC_PATH=/snap/lxd/current/bin/lxdLXD_DIR=/var/snap/lxd/common/lxd/LXD_SHIFTFS=falseSNAP_REVISION=12317TMPDIR=/tmpSNAP_CURRENT=/snap/lxd/currentXTABLES_LIBDIR=/snap/lxd/current/lib/xtables/JOURNAL_STREAM=9:67703SNAP_CONTEXT=7pPmVFdSD3LnKR6PllcwKxFCnYcjD37KHYy3vyrdDlVHSNAP_VERSION=3.18SNAP_INSTANCE_NAME=lxdPATH=/snap/lxd/12317/usr/sbin:/snap/lxd/12317/usr/bin:/snap/lxd/12317/sbin:/snap/lxd/12317/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/lxd/current/binINVOCATION_ID=c4ebf70a895645a4bbfb38093a771c1cXDG_RUNTIME_DIR=/run/user/0/snap.lxdLISTEN_FDNAMES=unixSNAP_DATA=/var/snap/lxd/12317LXD_CLUSTER_UPDATE=/snap/lxd/current/commands/refreshLANG=C.UTF-8SNAP_ARCH=amd64SNAP_USER_COMMON=/root/snap/lxd/commonSNAP_COOKIE=7pPmVFdSD3LnKR6PllcwKxFCnYcjD37KHYy3vyrdDlVHSNAP_REEXEC=SNAP_NAME=lxdLXD_LXC_HOOK=/snap/lxd/current/lxc/hooks/PWD=/var/snap/lxd/12317SNAP=/snap/lxd/12317
Still don’t see any reference to criu 
yes, that’s the problem.
I have found the start script for lxd in /snap/lxd/current/commands/daemon.start, it contains this:
# Setup CRIU
if [ "${criu_enable:-"false"}" = "true" ]; then
echo "==> Enabling CRIU"
export PATH="${SNAP_CURRENT}/criu:${PATH}"
fi
so there should be something in a log somewhere, let’s see… yes that’s it
sudo journalctl -u snap.lxd.daemon
you should see something about CRIU just after some systemd messages in your log. If not that’s because criu.enable is not equal to ‘true’ and if you have set it up correctly it could be snap’s fault if the lxd start script don’t see it.
So looking in the log, there is nothing about CRIU mentioned. It would seem that for some reason the daemon startup script is failing to see that criu is enabled.
I tried something as a total hack to try to confirm this theory…
I copied the contents /snap/lxd/current/commands to ~/tmp and then did this:
sudo mount --bind -o nodev,ro /home/$USER/tmp /snap/lxd/current/commands/
Then I modified the daemon.start in the ~/tmp directory …
# Setup CRIU
# if [ "${criu_enable:-"false"}" = "true" ]; then
echo "==> Enabling CRIU"
export PATH="${SNAP_CURRENT}/criu:${PATH}"
# fi
… and I restarted the daemon. Sure enough, the log now shows CRIU being enabled. Next, I tried to create a stateful snapshot. This time, the error about CRIU not bieng installed was gone. There was in fact another error that seems unrelated (Error: mkdir /var/snap/lxd/common/lxd/containers/dev-dwk: file exists) but I’m not worried about that since obviously this is a total hack for testing only.
The main thing I wanted to prove is that for some reason, the LXD startup is not seeing the configuration key.
Any thoughts about where to go from here?
Seems convincing, that’s the main point.
Not immediately. Anyway I’m going to sleep soon.
If in 24 hours you have no further help (from me or someone else) in your place I’d think of backup your containers / uninstall lxd and snap itself and install again, restore your containers if necessary and try again. There seems to be something wrong in the snap / lxd interaction, maybe something is broken.
It will be a while before I’ll be able to test out the approach. Marking the thread as resolved for now. I’ll come back later with concrete results.
Just to confirm what worked for me after installing lxd with snap:
--> sudo snap set lxd criu.enable=true
--> sudo snap restart lxd
--> sudo snap get lxd criu
Key Value
criu.enable true
I no longer get the “CRIU not installed” error, but the dump does fail. I realize this is a CRIU error, but this is for a brand new launched ubuntu:20.04 image. If that can’t dump out of the box I doubt anything else can. But I will try some other images.
--> lxc snapshot test snap0 --stateful
Error: snapshot dump failed
(00.003747) Error (criu/namespaces.c:420): Can't dump nested uts namespace for 23202
(00.003750) Error (criu/namespaces.c:679): Can't make utsns id
(00.062213) Error (criu/util.c:618): exited, status=1
(00.125642) Error (criu/util.c:618): exited, status=1
(00.126361) Error (criu/cr-dump.c:1764): Dumping FAILED.
Yeah, I did that and confirmed the value. It still doesn’t work. What’s strange is that is I am working through another issue and it seems to me like values set with snap set lxd ... are simply not getting conveyed to the container on my machine. Trying to set debug=true so as to get the debug log messages is also not working for me. I’m scratching my head.
values set with
snap set lxd ...are simply not getting conveyed to the container on my machine
By the container do you mean the snap “container” that has LXD installed in or the LXD container you are trying to snapshot? The configuring part is really about configuring your LXD installation and not the container you are trying to snapshot.
Sorry, bad word choice. I meant that the snap doesn’t seem to see the value. I hacked the snap a while back and got into the file system and manually changed criu.enabled to true, and then the snap “saw” CRIU as enabled (although I got a different error message at that point). But setting criu.enable=true with sudo snap set lxd criu.enable=true never seemed to get the snap to recognize CRIU as enabled. snap get lxd criu will show it as criu.enable true though.
Ya its a bit weird with snap. I don’t really like them to be honest considering they have all the monitors and configuration loading blah. Systemd is way more than enough of this kind of stuff anyways.