Recovering containers from crashed HD

LXD
,
1

My Ubuntu server (with 4 HDs) was running lxc and my containers, but the HD with the operating system crashed. I installed Ubuntu again in a new HD and now, lxc is running but it doesn’t know about the existence of my old container (obviously):

fmello@server2:~$ lxc list
±-----±------±-----±-----±-----±----------+
| NAME | STATE | IPV4 | IPV6 | TYPE | SNAPSHOTS |
±-----±------±-----±-----±-----±----------+

I can see all my old containers by browsing at the filesystem, because the storage pool is mounted:

fmello@server2:/mnt/@/var/lib/lxd/storage-pools/default/containers$ ls
ai2AIDevel ai2AIProd ai2Base ai2CA ai2GitLab ai2Web2 boomerang julia raves s3msPub s3msPub2

How do I make the new lxc installation see and run those old containers?

2

Did you try using lxd recover and the disaster recovery instructions at Backups | LXD ?

3

I tried lxd recover as my initial option. But it fails to run:

fmello@fmello:/var/snap/lxd/common/lxd$ lxd recover
Error: unknown command “recover” for “lxd”

4

What version of LXD are you running?

5

Both LXD and LXC are 4.0.7

6

Ah okay, for 4.0 LTS, lxd recover isn’t available yet (will be in 4.0.8), so you’d need to use lxd import instead following Backups | LXD

7

So, I downgraded lxd in order to use lxd recover:

sudo snap remove lxd
sudo snap install lxd --channel=3.0/stable

Now lxd version is 3.0.4, but something strange happens when I try to start recovery

fmello@fmello:~$ sudo lxd recover
EROR[09-15|16:20:53] Failed to start the daemon: LXD is already running
Error: LXD is already running

8

lxd recover was added with LXD 4.17 so your downgrade did the complete opposite of what you’d need to do to get it :slight_smile:

If you’re using anything prior to LXD 4.17, you’ll need to use lxd import instead as covered in the documentation I linked you to.

(The error you’re getting on 3.0 is because that release has a bug where lxd whatever is considered to be the same as starting LXD)

9

OK

sudo snap remove lxd
sudo snap install lxd

… and I am back with lxd 4.18

I read the following documentations, but they are superficial, doesn’t explain lxd recover usage:
Backing up a LXD server
Disaster recovery

I ran the lxd recover, but I don’t know how to indicate that the files from my old storage pool are already available at /mnt/@/var/lib/lxd/storage-pools (which is the file system partition I mounted from /dev/sdb2)

10

Add a new pool through lxd recover matching the name of the pool you’d like to import, select btrfs as the backend and put /mnt/@/var/lib/lxd/storage-pools/NAME as the source (updating NAME to match your setup).

11

fmello@server2:/mnt/@/var/lib/lxd/storage-pools$ sudo ls
default

fmello@server2:/mnt/@/var/lib/lxd/storage-pools$ sudo lxd recover
This LXD server currently has the following storage pools:
Would you like to recover another storage pool? (yes/no) [default=no]: yes
Name of the storage pool: default
Name of the storage backend (lvm, zfs, ceph, btrfs, cephfs, dir): btrfs
Source of the storage pool (block device, volume group, dataset, path, … as applicable): /mnt/@/var/lib/lxd/storage-pools/default
Additional storage pool configuration property (KEY=VALUE, empty when done):
Would you like to recover another storage pool? (yes/no) [default=no]:
The recovery process will be scanning the following storage pools:

  • NEW: “default” (backend=“btrfs”, source=“/mnt/@/var/lib/lxd/storage-pools/default”)
    Would you like to continue with scanning for lost volumes? (yes/no) [default=yes]:
    Scanning for unknown volumes…
    Error: Failed validation request: Failed mounting pool “default”: Failed to create directory “/var/snap/lxd/common/lxd/storage-pools/default/custom”: mkdir /var/snap/lxd/common/lxd/storage-pools/default/custom: read-only file system

This directory that is read-only, is and area from lxd package. Why it is read-only if lxd need to write on it?

12

It suggests that /mnt/@/var/lib/lxd/storage-pools/default is read-only or otherwise unavailable from within the snap.

Can you show ls -lh /mnt/@/var/lib/lxd/storage-pools/default?

13

fmello@server2:~$ ls -lh /mnt/@/var/lib/lxd/storage-pools/default
total 0
drwxr-xr-x 1 root root 162 Sep 9 21:18 containers
drwxr-xr-x 1 root root 0 Jun 18 2019 custom
drwxr-xr-x 1 root root 0 Jul 25 03:47 images
drwxr-xr-x 1 root root 60 Sep 12 13:52 snapshots

14

Can you show ls -lh /var/snap/lxd/common/mntns/mnt/?

15

fmello@server2:~$ ls -lh /var/snap/lxd/common/mntns/mnt/
ls: cannot access ‘/var/snap/lxd/common/mntns/mnt/’: No such file or directory

fmello@server2:/var/snap/lxd/common$ ls -lh
total 28K
-rw-r–r-- 1 root root 284 Sep 15 17:20 config
drwxr-xr-x 2 root root 4.0K Sep 15 17:22 global-conf
drwxr-xr-x 2 root root 4.0K Sep 15 17:22 lxc
drwx–x–x 17 root root 4.0K Sep 15 18:39 lxd
lrwxrwxrwx 1 root root 16 Sep 15 17:22 mntns → /proc/29783/root
drwx------ 2 root root 4.0K Sep 15 17:22 ns
drwx–x–x 2 root root 4.0K Sep 15 17:22 shmounts
-rw-r–r-- 1 root root 0 Sep 15 18:36 state
drwxr-xr-x 3 root root 4.0K Sep 15 17:22 var

16

Can you show cat /proc/mounts?

17

fmello@server2:/proc$ cat /proc/mounts
sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0
proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0
udev /dev devtmpfs rw,nosuid,noexec,relatime,size=82323984k,nr_inodes=20580996,mode=755 0 0
devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0
tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=16473828k,mode=755 0 0
/dev/mapper/ubuntu–vg–1-ubuntu–lv / ext4 rw,relatime 0 0
securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0
tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0
tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0
tmpfs /sys/fs/cgroup tmpfs ro,nosuid,nodev,noexec,mode=755 0 0
cgroup2 /sys/fs/cgroup/unified cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate 0 0
cgroup /sys/fs/cgroup/systemd cgroup rw,nosuid,nodev,noexec,relatime,xattr,name=systemd 0 0
pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0
efivarfs /sys/firmware/efi/efivars efivarfs rw,nosuid,nodev,noexec,relatime 0 0
none /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0
cgroup /sys/fs/cgroup/hugetlb cgroup rw,nosuid,nodev,noexec,relatime,hugetlb 0 0
cgroup /sys/fs/cgroup/blkio cgroup rw,nosuid,nodev,noexec,relatime,blkio 0 0
cgroup /sys/fs/cgroup/freezer cgroup rw,nosuid,nodev,noexec,relatime,freezer 0 0
cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0
cgroup /sys/fs/cgroup/memory cgroup rw,nosuid,nodev,noexec,relatime,memory 0 0
cgroup /sys/fs/cgroup/pids cgroup rw,nosuid,nodev,noexec,relatime,pids 0 0
cgroup /sys/fs/cgroup/perf_event cgroup rw,nosuid,nodev,noexec,relatime,perf_event 0 0
cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0
cgroup /sys/fs/cgroup/cpuset cgroup rw,nosuid,nodev,noexec,relatime,cpuset 0 0
cgroup /sys/fs/cgroup/devices cgroup rw,nosuid,nodev,noexec,relatime,devices 0 0
cgroup /sys/fs/cgroup/rdma cgroup rw,nosuid,nodev,noexec,relatime,rdma 0 0
systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=28,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=84998 0 0
hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0
mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0
debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0
tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0
fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0
configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0
/dev/loop0 /snap/core/11606 squashfs ro,nodev,relatime 0 0
/dev/sda2 /boot ext4 rw,relatime 0 0
/dev/loop1 /snap/core20/1081 squashfs ro,nodev,relatime 0 0
/dev/sda1 /boot/efi vfat rw,relatime,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro 0 0
/dev/loop2 /snap/core18/2128 squashfs ro,nodev,relatime 0 0
/dev/loop4 /snap/snapd/12704 squashfs ro,nodev,relatime 0 0
/dev/loop3 /snap/lxd/21497 squashfs ro,nodev,relatime 0 0
/dev/loop5 /snap/snapd/12883 squashfs ro,nodev,relatime 0 0
tmpfs /run/snapd/ns tmpfs rw,nosuid,nodev,noexec,relatime,size=16473828k,mode=755 0 0
nsfs /run/snapd/ns/lxd.mnt nsfs rw 0 0
tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=16473828k,mode=700,uid=1000,gid=1000 0 0
/dev/sdb4 /mnt btrfs rw,relatime,space_cache,subvolid=5,subvol=/ 0 0

18

Is there anything other than that LXD pool on /dev/sdb4?

19

The /dev/sdb4 is the drive my old OS mounted the /. I manage to mount this driver at /mnt. This is the ls of this path:

fmello@server2:/mnt/@$ ls
ai2esb bk cdrom data0 desenv dev home initrd.img.old lib64 mnt proc run snap sys usr vmlinuz bin boot clientdataset data1 desenvdata etc initrd.img lib media opt root sbin srv tmp var vmlinuz.old

20

Ok and what do you intend to do with the rest of the stuff on there?
Do you just intend to keep it mounted at /mnt indefinitely?