My boot drive on my server is in a failing state. I was able to successfully dd the entirety of the /var/snap/lxd/* dir. Since I wanted to upgrade to an nvme drive I went ahead and did an mobo/cpu upgrade as well(not sure if this would affect anything).
I then did a fresh install of ubuntu server and ran through the following steps to attempt to restore the snap lxd install.
I’m at a loss at where to go from here. If I could recover just the containers from the recovered snap install and start everything else from fresh I would be content. That being said I still have access to the failing drive and can likely boot to it and access the entirety of the original lxd install.
Thoughts?
Thanks,
Jonathon
EDITS:
Error: Get http://unix.socket/1.0: dial unix /var/lib/lxd/unix.socket: connect: no such file or directory
$ sudo lxd
WARN[09-11|00:03:16] CGroup memory swap accounting is disabled, swap limits will be ignored.
EROR[09-11|00:03:16] Failed to start the daemon: ZFS storage pool "lxd_zfs" could not be imported:
Error: ZFS storage pool "lxd_zfs" could not be imported:
$ sudo systemctl status lxd
Unit lxd.service could not be found.
$ sudo systemctl status snap.lxd.activate.service
● snap.lxd.activate.service - Service for snap application lxd.activate
Loaded: loaded (/etc/systemd/system/snap.lxd.activate.service; enabled; vendor preset: enabled)
Active: inactive (dead) since Tue 2019-09-10 23:52:23 MDT; 11min ago
Process: 21393 ExecStart=/usr/bin/snap run lxd.activate (code=exited, status=0/SUCCESS)
Main PID: 21393 (code=exited, status=0/SUCCESS)
Sep 10 23:52:23 homeserver systemd[1]: Starting Service for snap application lxd.activate...
Sep 10 23:52:23 homeserver lxd.activate[21393]: => Starting LXD activation
Sep 10 23:52:23 homeserver lxd.activate[21393]: ==> Loading snap configuration
Sep 10 23:52:23 homeserver lxd.activate[21393]: ==> Checking for socket activation support
Sep 10 23:52:23 homeserver lxd.activate[21393]: ==> Setting LXD socket ownership
Sep 10 23:52:23 homeserver lxd.activate[21393]: ==> LXD never started on this system, no need to start it now
Sep 10 23:52:23 homeserver systemd[1]: Started Service for snap application lxd.activate.
$ sudo systemctl status snap.lxd.
snap.lxd.activate.service snap.lxd.daemon.service snap.lxd.daemon.unix.socket
$ sudo systemctl status snap.lxd.daemon.service
● snap.lxd.daemon.service - Service for snap application lxd.daemon
Loaded: loaded (/etc/systemd/system/snap.lxd.daemon.service; static; vendor preset: enabled)
Active: failed (Result: exit-code) since Tue 2019-09-10 23:52:42 MDT; 11min ago
Process: 23219 ExecStart=/usr/bin/snap run lxd.daemon (code=exited, status=1/FAILURE)
Main PID: 23219 (code=exited, status=1/FAILURE)
Sep 10 23:52:42 homeserver systemd[1]: snap.lxd.daemon.service: Service hold-off time over, scheduling restart.
Sep 10 23:52:42 homeserver systemd[1]: snap.lxd.daemon.service: Scheduled restart job, restart counter is at 10.
Sep 10 23:52:42 homeserver systemd[1]: Stopped Service for snap application lxd.daemon.
Sep 10 23:52:42 homeserver systemd[1]: snap.lxd.daemon.service: Start request repeated too quickly.
Sep 10 23:52:42 homeserver systemd[1]: snap.lxd.daemon.service: Failed with result 'exit-code'.
Sep 10 23:52:42 homeserver systemd[1]: Failed to start Service for snap application lxd.daemon.
$ sudo systemctl status snap.lxd.daemon.unix.socket
● snap.lxd.daemon.unix.socket - Socket unix for snap application lxd.daemon
Loaded: loaded (/etc/systemd/system/snap.lxd.daemon.unix.socket; enabled; vendor preset: enabled)
Active: failed (Result: service-start-limit-hit) since Tue 2019-09-10 23:52:42 MDT; 11min ago
Listen: /var/snap/lxd/common/lxd/unix.socket (Stream)
Sep 09 08:32:10 homeserver systemd[1]: Listening on Socket unix for snap application lxd.daemon.
Sep 10 23:52:42 homeserver systemd[1]: snap.lxd.daemon.unix.socket: Failed with result 'service-start-limit-hit'.
$
$
$ sudo cat /var/snap/lxd/common/lxd/logs/lxd.log
t=2019-09-10T23:52:41-0600 lvl=info msg="LXD 3.17 is starting in normal mode" path=/var/snap/lxd/common/lxd
t=2019-09-10T23:52:41-0600 lvl=info msg="Kernel uid/gid map:"
t=2019-09-10T23:52:41-0600 lvl=info msg=" - u 0 0 4294967295"
t=2019-09-10T23:52:41-0600 lvl=info msg=" - g 0 0 4294967295"
t=2019-09-10T23:52:41-0600 lvl=info msg="Configured LXD uid/gid map:"
t=2019-09-10T23:52:41-0600 lvl=info msg=" - u 0 1000000 1000000000"
t=2019-09-10T23:52:41-0600 lvl=info msg=" - g 0 1000000 1000000000"
t=2019-09-10T23:52:41-0600 lvl=warn msg="CGroup memory swap accounting is disabled, swap limits will be ignored."
t=2019-09-10T23:52:41-0600 lvl=info msg="Kernel features:"
t=2019-09-10T23:52:41-0600 lvl=info msg=" - netnsid-based network retrieval: no"
t=2019-09-10T23:52:41-0600 lvl=info msg=" - uevent injection: no"
t=2019-09-10T23:52:41-0600 lvl=info msg=" - seccomp listener: no"
t=2019-09-10T23:52:41-0600 lvl=info msg=" - unprivileged file capabilities: yes"
t=2019-09-10T23:52:41-0600 lvl=info msg=" - shiftfs support: no"
t=2019-09-10T23:52:41-0600 lvl=info msg="Initializing local database"
t=2019-09-10T23:52:41-0600 lvl=info msg="Starting /dev/lxd handler:"
t=2019-09-10T23:52:41-0600 lvl=info msg=" - binding devlxd socket" socket=/var/snap/lxd/common/lxd/devlxd/sock
t=2019-09-10T23:52:41-0600 lvl=info msg="REST API daemon:"
t=2019-09-10T23:52:41-0600 lvl=info msg=" - binding Unix socket" inherited=true socket=/var/snap/lxd/common/lxd/unix.socket
t=2019-09-10T23:52:41-0600 lvl=info msg="Initializing global database"
t=2019-09-10T23:52:41-0600 lvl=info msg="Initializing storage pools"
t=2019-09-10T23:52:41-0600 lvl=eror msg="Failed to start the daemon: ZFS storage pool \"lxd_zfs\" could not be imported: "
t=2019-09-10T23:52:41-0600 lvl=info msg="Starting shutdown sequence"
t=2019-09-10T23:52:41-0600 lvl=info msg="Stopping REST API handler:"
t=2019-09-10T23:52:41-0600 lvl=info msg=" - closing socket" socket=/var/snap/lxd/common/lxd/unix.socket
t=2019-09-10T23:52:41-0600 lvl=info msg="Stopping /dev/lxd handler:"
t=2019-09-10T23:52:41-0600 lvl=info msg=" - closing socket" socket=/var/snap/lxd/common/lxd/devlxd/sock
t=2019-09-10T23:52:41-0600 lvl=info msg="Closing the database"
t=2019-09-10T23:52:41-0600 lvl=info msg="Unmounting temporary filesystems"
t=2019-09-10T23:52:41-0600 lvl=info msg="Done unmounting temporary filesystems```
As it stands I’m still getting the cannot impot 'lxd_zfs': no such pool availble
I’m not sure if this is due to the fact that my lxd_zfs dir is located within the /var/snap/lxd/common/lxd/storage-pools/ dir or not.
Here is also the included dir structure in case it may provide anymore valuable insight.
You are correct the lxd-client was not purged from the dep package. Per your linked post I went through the following steps:
sudo snap stop lxd
sudo snap remove lxd
sudo apt --purge --auto-remove lxd-client
sudo apt --purge --auto-remove lxd
sudo apt autoremove
sudo snap install lxd
sudo lxd init
lxc list
# Success, LXC is working again.
# Now to try and replace the snap install with my backed up version
sudo snap stop lxd
sudo cp -R /mnt/extracted-lxd-dir/var/snap/lxd/common/lxd/ /var/sna/lxd/common/lxd/
sudo snap start lxd
lxc list
# LXC fails again
# Output appears to be the same but now referencing the proper install location
# Error: Get http://unix.socket/1.0: dial unix /var/snap/lxd/common/lxd/unix.socket: connect: no such file or directory
@stgraber here is the requested output from the above commands. Something to take note is that I do not have a snap.lxd.daemon.unix.socket. This is what’s available from my autocomplete for the snap.lxd. journals.
Ok, you should try systemctl restart snap.lxd.daemon.unix.socket that should get the socket file back online. Then you can try lxc list see if maybe the daemon feels like starting this time.
Your log indicates that last startup attempt was about a week ago and the daemon wasn’t happy due to a missing ZFS pool, but maybe that got corrected since.
Doesn’t appear to work. I think at this point I’m best off just starting from scratch and trying to boot from the failing drive to make the appropriate images of all my containers. That lxd_zfs pool was created on my old install via lxd init and so I was expecting to just be able to drop the old install in on the new HDD but apparently that’s not quite the case.
Depends on what was backing that pool. If it was a file backed pool and you have the file in /var/snap/lxd/common/lxd/disks then that should be fine.
If it was backed by some physical disk/partition, then just restoring the content of /var/snap/lxd isn’t enough, you also need to get that zpool back in place somehow.