Hi I just recovered from a snapd crash that prevented me from accessing my containers.
I could have lost all my container data because of snapd.
I would like to migrate all my container and their config files including profile config files outside of the snap directory. I am able to use zfs send/receive to copy individual containers. This is not exactly what I want. I want to have all the lxd data + containers outside of the snap environment. Let snapd manage only snap related issues. So that if I have to purge the lxd snap I sill have my data so that when I reinstall lxd I can use my containers. I also would like to be able to access the rootfs and the config files without being dependent on the lxd.daemon.
The other issues is whether I can do the migration safely (i.e. moving the data and deleting the old image)
Snapd itself only stores stuff in /var/lib/snapd, everything under /var/snap/lxd is in fact managed by LXD, not snapd.
The most critical part of this is /var/snap/lxd/common/lxd which stores everything that LXD uses to run. You can technically move that onto a separate partition and bind-mount it into place, but that won’t really change anything.
What actually happened in your case? snapd never touches /var/snap/lxd/common/* during a refresh, so I’m a bit confused as to what you experience.
The closest thing to what you’re describing would be a LXD database issue which would then show up as a repeat crash of LXD on startup, but that’s not a snapd issue, it’s a LXD or hardware issue and that can normally be recovered from very easily if you report the issue here when it happens
I upgraded ubuntu to 20.10. Because I am also using syncoid for backup zsys stopped working after the upgrade. I removed all the syncoid snapshots with the -R option,which actually destroyed my system. I was able to recover to 20.04 from snapshots, but snapd stopped working (for all the snaps)… I basically lost access to my containers because of snapd. I had to used a backup of my whole system from 2 weeks ago to recover. I also had to install zfs from scratch. If I had an open access to the lxd container’s. (like in lxc in /var/lib/lxc). I could have recovered a lot more. I do not like not being able to access the container and config file. I basically want to be able to access everything in rootfs +config file from the host computer.
My solution is not very good. I used send/receive to create a separate pool of my main container. Now I need to be able to tell lxd to dump the old pool and use the new pool. I am not sure how to do it safely.
Ok, we’ve seen a couple of folks get into a similar situation where you effectively get snapd complaining about missing systemd units and not being able to start stuff.
The easy way to recover in such cases is to move /var/snap/lxd/common/lxd somewhere safe, remove the LXD snap, reboot the system, install the LXD snap again, stop it, move the data back in place and then start it back up.
That’s actually pretty much exactly what I told you in:
No It did not work. I had a chromium snap. I tried many things with it including removal . it is not work. (I did not want to risk the lxd snap) Even core and core18 of snapd had issues. I am simplifying the story. During the recovery period. There was a time that I was not able to boot, but I did have access to the data pools. This link was very helpful in restoring my system. [https://openzfs.github.io/openzfs-docs/Getting%20Started/Ubuntu/Ubuntu%2020.04%20Root%20on%20ZFS.html]
Back to my original question. Can I migrate at least one container from the lxd storage area to a separate zfs pool.
Ah, okay, if you had reported that, I’d have told you to just manually get onto a clean state with:
rm -rf /var/lib/snapd/*
reboot
rm -rf /snap/*
snap install lxd
snap stop lxd
Move back /var/snap/lxd/common/lxd into place
snap start lxd
LXD can only use ZFS pools that it manages. They can be backed by external disk/partitions, but you cannot just tell it to load an existing container from a random pool.
There is support for disaster recovery which allows rebuilding some of the LXD config entries from a re-attached storage pool, but this requires the storage pool to be the same (name and location) as the instance was created on.
Regarding the file lxd.img, which is clearly where everything is in. Is there any way where I can mount it or probe it to see what is inside. Now that my system restored I am missing 2 weeks worth of data. I can restore the lxd.img with the missing data from snapshot to a temporary directory. But then how do I get my data.
