Hi,
I don’t know what happened but snap and lxd installations were broken somehow in my Debian Stretch. I’ve managed to restore the snap and lxd but now i can’t see my containers with lxc list command.
The weird thing is containers are running.(The apps inside the containers are running) All files under “/var/snap/lxd/common/lxd/containers” are lost and folder is empty. How can i restore containers ?
Thanks.
Do you perhaps have both the deb and snap installed?
Oh, on stretch, so that’s not super likely.
What does find /var/snap/lxd/common/mntns/var/snap/lxd/common/lxd get you?
root@server:~# find /var/snap/lxd/common/mntns/var/snap/lxd/common/lxd
/var/snap/lxd/common/mntns/var/snap/lxd/common/lxd
/var/snap/lxd/common/mntns/var/snap/lxd/common/lxd/backups
/var/snap/lxd/common/mntns/var/snap/lxd/common/lxd/devices
/var/snap/lxd/common/mntns/var/snap/lxd/common/lxd/snapshots
/var/snap/lxd/common/mntns/var/snap/lxd/common/lxd/cache
/var/snap/lxd/common/mntns/var/snap/lxd/common/lxd/cache/instance_types.yaml
/var/snap/lxd/common/mntns/var/snap/lxd/common/lxd/storage-pools
/var/snap/lxd/common/mntns/var/snap/lxd/common/lxd/database
/var/snap/lxd/common/mntns/var/snap/lxd/common/lxd/database/local.db
/var/snap/lxd/common/mntns/var/snap/lxd/common/lxd/database/global
/var/snap/lxd/common/mntns/var/snap/lxd/common/lxd/database/global/open-2
/var/snap/lxd/common/mntns/var/snap/lxd/common/lxd/database/global/open-3
/var/snap/lxd/common/mntns/var/snap/lxd/common/lxd/database/global/1-1
/var/snap/lxd/common/mntns/var/snap/lxd/common/lxd/database/global/db.bin
/var/snap/lxd/common/mntns/var/snap/lxd/common/lxd/database/global/db.bin-wal
/var/snap/lxd/common/mntns/var/snap/lxd/common/lxd/database/global/metadata1
/var/snap/lxd/common/mntns/var/snap/lxd/common/lxd/database/global/open-1
/var/snap/lxd/common/mntns/var/snap/lxd/common/lxd/database/global/metadata2
/var/snap/lxd/common/mntns/var/snap/lxd/common/lxd/database/global/2-16
/var/snap/lxd/common/mntns/var/snap/lxd/common/lxd/unix.socket
/var/snap/lxd/common/mntns/var/snap/lxd/common/lxd/disks
/var/snap/lxd/common/mntns/var/snap/lxd/common/lxd/networks
/var/snap/lxd/common/mntns/var/snap/lxd/common/lxd/security
/var/snap/lxd/common/mntns/var/snap/lxd/common/lxd/devlxd
/var/snap/lxd/common/mntns/var/snap/lxd/common/lxd/devlxd/sock
/var/snap/lxd/common/mntns/var/snap/lxd/common/lxd/logs
/var/snap/lxd/common/mntns/var/snap/lxd/common/lxd/logs/lxd.log.1
/var/snap/lxd/common/mntns/var/snap/lxd/common/lxd/logs/lxd.log
/var/snap/lxd/common/mntns/var/snap/lxd/common/lxd/server.key
/var/snap/lxd/common/mntns/var/snap/lxd/common/lxd/shmounts
/var/snap/lxd/common/mntns/var/snap/lxd/common/lxd/images
/var/snap/lxd/common/mntns/var/snap/lxd/common/lxd/containers
/var/snap/lxd/common/mntns/var/snap/lxd/common/lxd/server.crtIt’s snap package only.
Ok, that looks pretty empty indeed.
Can you show ps fauxww so I can see that running container too?
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
root 3613 0.0 0.0 295332 96 ? Ss Jan07 0:00 [lxc monitor] /var/snap/lxd/common/lxd/containers xxxx
root 3644 0.0 0.0 225052 2088 ? Ss Jan07 0:05 \_ /sbin/init
root 3705 0.0 0.0 110312 2240 ? S<s Jan07 0:08 \_ /lib/systemd/systemd-journald
systemd+ 3773 0.0 0.0 80040 1392 ? Ss Jan07 0:03 \_ /lib/systemd/systemd-networkd
systemd+ 3777 0.0 0.0 70756 788 ? Ss Jan07 2:37 \_ /lib/systemd/systemd-resolved
root 3802 0.0 0.0 287992 888 ? Ssl Jan07 0:24 \_ /usr/lib/accountsservice/accounts-daemon
systemd+ 3806 0.0 0.0 50036 688 ? Ss Jan07 0:00 \_ /usr/bin/dbus-daemon --system --address=systemd: --nofork --nopidfile --systemd-activation --syslog-only
root 3812 0.0 0.0 170828 8000 ? Ssl Jan07 0:00 \_ /usr/bin/python3 /usr/bin/networkd-dispatcher --run-startup-triggers
111 3818 0.0 0.0 8304 124 ? Ss Jan07 0:48 \_ /usr/sbin/vnstatd -n
daemon 3819 0.0 0.0 28328 348 ? Ss Jan07 0:00 \_ /usr/sbin/atd -f
root 3821 0.0 0.0 31744 664 ? Ss Jan07 0:02 \_ /usr/sbin/cron -f
systemd+ 3822 0.0 0.0 267268 980 ? Ssl Jan07 0:00 \_ /usr/sbin/rsyslogd -n
root 3823 0.0 0.0 62108 660 ? Ss Jan07 0:01 \_ /lib/systemd/systemd-logind
root 3825 0.0 0.0 4624 0 ? Ss Jan07 0:00 \_ /bin/sh -c PLEX_MEDIA_SERVER_INFO_VENDOR="$(grep ^NAME= /etc/os-release | awk -F= "{print \$2}" | tr -d \" )" PLEX_MEDIA_SERVER_INFO_DEVICE="PC" PLEX_MEDIA_SERVER_INFO_MODEL="$(uname -m)" PLEX_MEDIA_SERVER_INFO_PLATFORM_VERSION="$(grep ^VERSION= /etc/os-release | awk -F= "{print \$2}" | tr -d \" )" LD_LIBRARY_PATH=/usr/lib/plexmediaserver/lib "/usr/lib/plexmediaserver/Plex Media Server"
root 3863 9.2 20.6 8326152 6786684 ? Sl Jan07 2164:35 | \_ /usr/lib/plexmediaserver/Plex Media Server
root 3944 0.3 1.9 2576916 650784 ? SNl Jan07 76:23 | \_ Plex Plug-in [com.plexapp.system] /usr/lib/plexmediaserver/Resources/Plug-ins-349e9837e/Framework.bundle/Contents/Resources/Versions/2/Python/bootstrap.py --server-version 1.18.3.2156-349e9837e /usr/lib/plexmediaserver/Resources/Plug-ins-349e9837e/System.bundle
root 4030 0.0 0.0 440440 156 ? Sl Jan07 0:33 | \_ /usr/lib/plexmediaserver/Plex Tuner Service /usr/lib/plexmediaserver/Resources/Tuner/Private /usr/lib/plexmediaserver/Resources/Tuner/Shared 1.18.3.2156-349e9837e 32600 /waitmutex
root 32831 0.1 0.0 18388 1220 ? S Jan07 43:45 | \_ Plex EAE Service
root 22364 1.1 0.1 1030816 43328 ? Sl 09:48 0:11 | \_ Plex Plug-in [com.plexapp.agents.localmedia] /usr/lib/plexmediaserver/Resources/Plug-ins-349e9837e/Framework.bundle/Contents/Resources/Versions/2/Python/bootstrap.py --server-version 1.18.3.2156-349e9837e /usr/lib/plexmediaserver/Resources/Plug-ins-349e9837e/LocalMedia.bundle
root 22484 0.4 0.1 1305580 41532 ? Sl 09:48 0:04 | \_ Plex Plug-in [com.plexapp.agents.themoviedb] /usr/lib/plexmediaserver/Resources/Plug-ins-349e9837e/Framework.bundle/Contents/Resources/Versions/2/Python/bootstrap.py --server-version 1.18.3.2156-349e9837e /usr/lib/plexmediaserver/Resources/Plug-ins-349e9837e/TheMovieDB.bundle
root 22524 0.5 0.1 1437932 48052 ? Sl 09:48 0:05 | \_ Plex Plug-in [com.plexapp.agents.thetvdb] /usr/lib/plexmediaserver/Resources/Plug-ins-349e9837e/Framework.bundle/Contents/Resources/Versions/2/Python/bootstrap.py --server-version 1.18.3.2156-349e9837e /usr/lib/plexmediaserver/Resources/Plug-ins-349e9837e/TheTVDB.bundle
root 22667 0.2 0.1 1101368 34952 ? Sl 09:48 0:02 | \_ Plex Plug-in [com.plexapp.agents.plexthememusic] /usr/lib/plexmediaserver/Resources/Plug-ins-349e9837e/Framework.bundle/Contents/Resources/Versions/2/Python/bootstrap.py --server-version 1.18.3.2156-349e9837e /usr/lib/plexmediaserver/Resources/Plug-ins-349e9837e/PlexThemeMusic.bundle
root 23963 0.0 0.0 39168 5944 ? S 10:00 0:00 | \_ /usr/lib/plexmediaserver/Plex Relay -p 443 -N -R 0:127.0.0.1:32401 -o StrictHostKeyChecking=yes -o UserKnownHostsFile=/var/lib/plexmediaserver/Library/Application Support/Plex Media Server/Cache/relayHostKey.txt -o NoneSwitch=yes -o NoneEnabled=yes -o LogLevel=VERBOSE -o PreferredAuthentications=password -o PubkeyAuthentication=no -l plexmania61 -F /dev/null 184.105.148.84
root 24116 8.4 0.1 306308 51364 ? Sl 10:01 0:17 | \_ /usr/lib/plexmediaserver/Plex Transcoder -codec:0 h264 -codec:1 eac3_eae -eae_prefix:1 b692ff0a4276d0f5-com-plexapp-android_ -noaccurate_seek -analyzeduration 20000000 -probesize 20000000 -i /home/plex/gdrive/TV/HOT/Star Trek - Picard/Season 01/Star Trek - Picard.S01E01.Episode 1 WEBDL-1080p.NTb.mkv -filter_complex [0:1] aresample=async=1:ocl='5.1':osr=48000[0] -map 0:0 -codec:0 copy -map [0] -metadata:s:1 language=eng -codec:1 aac -strict:1 experimental -aac_coder:1 fast -q:1 0 -segment_format mpegts -f ssegment -individual_header_trailer 0 -segment_time 10 -segment_start_number 0 -segment_copyts 1 -segment_time_delta 0.0625 -segment_list http://127.0.0.1:32400/video/:/transcode/session/b692ff0a4276d0f5-com-plexapp-android/c5b85a0b-ddd1-4d60-b882-80ce4f3fc253/seglist -segment_list_type csv -segment_list_size 5 -segment_list_separate_stream_times 1 -segment_list_unfinished 1 -max_delay 5000000 -avoid_negative_ts disabled -map_metadata -1 -map_chapters -1 media-%05d.ts -start_at_zero -copyts -vsync cfr -y -nostats -loglevel quiet -loglevel_plex error -progressurl http://127.0.0.1:32400/video/:/transcode/session/b692ff0a4276d0f5-com-plexapp-android/c5b85a0b-ddd1-4d60-b882-80ce4f3fc253/progress
root 3826 0.0 0.0 297780 11000 ? Ssl Jan07 8:00 \_ /usr/bin/python3 /usr/bin/fail2ban-server -xf start
root 3833 0.0 0.0 291444 1116 ? Ssl Jan07 0:00 \_ /usr/lib/policykit-1/polkitd --no-debug
root 3848 0.0 0.0 187668 8240 ? Ssl Jan07 0:00 \_ /usr/bin/python3 /usr/share/unattended-upgrades/unattended-upgrade-shutdown --wait-for-signal
root 3851 0.0 0.0 16408 296 pts/1 Ss+ Jan07 0:00 \_ /sbin/agetty -o -p -- \u --noclear --keep-baud console 115200,38400,9600 vt220
root 3862 0.0 0.0 72292 724 ? Ss Jan07 0:00 \_ /usr/sbin/sshd -D
113 3957 0.7 0.7 3031012 241160 ? Sl Jan07 184:29 \_ python /opt/tautulli/Tautulli.py --quiet --daemon --nolaunch --config /opt/tautulli/config.ini --datadir /opt/tautulli
root 37116 0.0 0.0 362276 248 ? Ss Jan09 0:00 [lxc monitor] /var/snap/lxd/common/lxd/containers zzzz
root 37149 0.0 0.0 159640 2856 ? Ss Jan09 0:04 \_ /sbin/init
root 37206 0.0 0.0 103440 1060 ? S<s Jan09 0:06 \_ /lib/systemd/systemd-journald
systemd+ 37271 0.0 0.0 80040 708 ? Ss Jan09 0:03 \_ /lib/systemd/systemd-networkd
systemd+ 37274 0.0 0.0 70760 1768 ? Ss Jan09 1:26 \_ /lib/systemd/systemd-resolved
root 37299 0.0 0.0 62112 1164 ? Ss Jan09 0:01 \_ /lib/systemd/systemd-logind
root 37302 0.0 0.0 287992 888 ? Ssl Jan09 0:21 \_ /usr/lib/accountsservice/accounts-daemon
111 37303 0.0 0.0 8304 124 ? Ss Jan09 0:41 \_ /usr/sbin/vnstatd -n
daemon 37304 0.0 0.0 28328 212 ? Ss Jan09 0:00 \_ /usr/sbin/atd -f
root 37308 0.0 0.0 170828 7424 ? Ssl Jan09 0:00 \_ /usr/bin/python3 /usr/bin/networkd-dispatcher --run-startup-triggers
systemd+ 37311 0.0 0.0 267264 828 ? Ssl Jan09 0:00 \_ /usr/sbin/rsyslogd -n
root 37314 0.0 0.0 31744 432 ? Ss Jan09 0:02 \_ /usr/sbin/cron -f
systemd+ 37316 0.0 0.0 50044 688 ? Ss Jan09 0:00 \_ /usr/bin/dbus-daemon --system --address=systemd: --nofork --nopidfile --systemd-activation --syslog-only
root 37322 0.0 0.0 297780 10968 ? Ssl Jan09 7:00 \_ /usr/bin/python3 /usr/bin/fail2ban-server -xf start
root 37324 0.0 0.0 72292 732 ? Ss Jan09 0:00 \_ /usr/sbin/sshd -D
root 37339 0.0 0.0 288876 836 ? Ssl Jan09 0:00 \_ /usr/lib/policykit-1/polkitd --no-debug
root 37341 0.0 0.0 187668 7908 ? Ssl Jan09 0:00 \_ /usr/bin/python3 /usr/share/unattended-upgrades/unattended-upgrade-shutdown --wait-for-signal
root 37345 0.0 0.0 16408 144 pts/0 Ss+ Jan09 0:00 \_ /sbin/agetty -o -p -- \u --noclear --keep-baud console 115200,38400,9600 vt220
112 37430 0.7 0.7 2950732 239540 ? Sl Jan09 147:42 \_ python /opt/tautulli/Tautulli.py --quiet --daemon --nolaunch --config /opt/tautulli/config.ini --datadir /opt/tautulli
root 38062 0.0 0.0 4624 0 ? Ss Jan09 0:00 \_ /bin/sh -c PLEX_MEDIA_SERVER_INFO_VENDOR="$(grep ^NAME= /etc/os-release | awk -F= "{print \$2}" | tr -d \" )" PLEX_MEDIA_SERVER_INFO_DEVICE="PC" PLEX_MEDIA_SERVER_INFO_MODEL="$(uname -m)" PLEX_MEDIA_SERVER_INFO_PLATFORM_VERSION="$(grep ^VERSION= /etc/os-release | awk -F= "{print \$2}" | tr -d \" )" LD_LIBRARY_PATH=/usr/lib/plexmediaserver/lib "/usr/lib/plexmediaserver/Plex Media Server"
root 38072 7.2 12.8 6461492 4234616 ? Sl Jan09 1473:42 \_ /usr/lib/plexmediaserver/Plex Media Server
root 38101 0.2 1.4 2516116 493516 ? SNl Jan09 54:16 \_ Plex Plug-in [com.plexapp.system] /usr/lib/plexmediaserver/Resources/Plug-ins-ac2afe5f8/Framework.bundle/Contents/Resources/Versions/2/Python/bootstrap.py --server-version 1.18.4.2171-ac2afe5f8 /usr/lib/plexmediaserver/Resources/Plug-ins-ac2afe5f8/System.bundle
root 38163 0.0 0.0 439708 160 ? Sl Jan09 0:29 \_ /usr/lib/plexmediaserver/Plex Tuner Service /usr/lib/plexmediaserver/Resources/Tuner/Private /usr/lib/plexmediaserver/Resources/Tuner/Shared 1.18.4.2171-ac2afe5f8 32600 /waitmutex
root 38373 0.0 0.0 947432 632 ? Sl Jan09 5:50 \_ Plex Plug-in [tv.plex.agents.music] /usr/lib/plexmediaserver/Resources/Plug-ins-ac2afe5f8/Framework.bundle/Contents/Resources/Versions/2/Python/bootstrap.py --server-version 1.18.4.2171-ac2afe5f8 /usr/lib/plexmediaserver/Resources/Plug-ins-ac2afe5f8/PlexMusic.bundle
root 29329 0.2 0.0 18388 1240 ? S Jan09 54:58 \_ Plex EAE Service
root 2561 0.8 0.1 958640 60080 ? Sl 08:42 0:43 \_ Plex Plug-in [com.plexapp.agents.localmedia] /usr/lib/plexmediaserver/Resources/Plug-ins-ac2afe5f8/Framework.bundle/Contents/Resources/Versions/2/Python/bootstrap.py --server-version 1.18.4.2171-ac2afe5f8 /usr/lib/plexmediaserver/Resources/Plug-ins-ac2afe5f8/LocalMedia.bundle
root 3278 1.0 0.3 1850420 117088 ? Sl 08:45 0:48 \_ Plex Plug-in [com.plexapp.agents.themoviedb] /usr/lib/plexmediaserver/Resources/Plug-ins-ac2afe5f8/Framework.bundle/Contents/Resources/Versions/2/Python/bootstrap.py --server-version 1.18.4.2171-ac2afe5f8 /usr/lib/plexmediaserver/Resources/Plug-ins-ac2afe5f8/TheMovieDB.bundle
root 22843 0.4 0.1 1372628 48644 ? Sl 09:49 0:04 \_ Plex Plug-in [com.plexapp.agents.thetvdb] /usr/lib/plexmediaserver/Resources/Plug-ins-ac2afe5f8/Framework.bundle/Contents/Resources/Versions/2/Python/bootstrap.py --server-version 1.18.4.2171-ac2afe5f8 /usr/lib/plexmediaserver/Resources/Plug-ins-ac2afe5f8/TheTVDB.bundle
root 22954 0.2 0.1 1027336 34964 ? Sl 09:49 0:02 \_ Plex Plug-in [com.plexapp.agents.plexthememusic] /usr/lib/plexmediaserver/Resources/Plug-ins-ac2afe5f8/Framework.bundle/Contents/Resources/Versions/2/Python/bootstrap.py --server-version 1.18.4.2171-ac2afe5f8 /usr/lib/plexmediaserver/Resources/Plug-ins-ac2afe5f8/PlexThemeMusic.bundle
root 57658 0.0 0.0 56660 708 ? Ss Jan16 0:00 /lib/systemd/systemd --user
root 57659 0.0 0.0 85128 68 ? S Jan16 0:00 \_ (sd-pam)
root 55327 0.0 0.0 944932 3924 ? Ssl 06:53 0:06 /usr/lib/snapd/snapd
root 56430 0.0 0.0 95436 0 ? Sl 06:59 0:00 lxcfs /var/snap/lxd/common/var/lib/lxcfs -p /var/snap/lxd/common/lxcfs.pid
root 59374 0.0 0.0 4500 120 ? Ss 07:22 0:00 /bin/sh /snap/lxd/12631/commands/daemon.start
root 59469 0.1 0.0 1452636 16016 ? Sl 07:22 0:10 \_ lxd --logfile /var/snap/lxd/common/lxd/logs/lxd.log --group lxdWhat does ls -lh /proc/3613/root/var/snap/lxd/common/lxd/storage-pools/ show you?
ls -lh /proc/3613/root/var/snap/lxd/common/lxd/storage-pools/
total 0Ok, that’s not looking so good. Make sure not to shut down the system as it may be exceedingly hard to recover things if you do.
Do you remember ever mounting some kind of partition over /var/snap/lxd or something like that?
This could explain everything vanishing all of a sudden, either because the data is now masked or because the data isn’t currently mounted.
Also, what storage backend was this on and was it backed by a file or a physical partition?
I didn’t mount anything over it. I remember python3 re-installed by another app. That could have caused all these issues.
It was btrfs backed by a file as i remember.
It could be more than enough for me to only recover the files. I can re-create containers if that could be possible.
And /var/snap/lxd folder was all empty before i recovered the lxd installation if that helps.
Can you show snap changes and also losetup -a?
I’m hoping that the loop pool is still available somewhere so you can at least dump that.
root@server:~# snap changes
ID Status Spawn Ready Summary
4 Done today at 06:40 EST today at 06:40 EST Refresh "lxd" snap
5 Done today at 06:50 EST today at 06:50 EST Refresh "core" snap from "edge" channel
6 Error today at 06:50 EST today at 06:50 EST Refresh "core" snap
7 Done today at 06:51 EST today at 06:51 EST Refresh "core" snap from "beta" channel
8 Done today at 06:52 EST today at 06:52 EST Refresh "core" snap
9 Done today at 06:58 EST today at 06:59 EST Install "lxd" snap from file "lxd_12631.snap"
10 Done today at 07:20 EST today at 07:20 EST Refresh "lxd" snap
11 Done today at 14:03 EST today at 14:03 EST Auto-refresh snap "lxd"
root@server:~# losetup -a
/dev/loop1: [2304]:7093713 (/var/lib/snapd/snaps/core_8268.snap (deleted))
/dev/loop6: [2304]:7093882 (/var/lib/snapd/snaps/lxd_13073.snap)
/dev/loop4: [2304]:7093614 (/var/lib/snapd/snaps/core_8509.snap)
/dev/loop2: [2304]:7094294 (/var/snap/lxd/common/lxd/disks/default.img (deleted))
/dev/loop0: [2304]:7094103 (/var/lib/snapd/snaps/lxd_11348.snap (deleted))
/dev/loop5: [2304]:7094073 (/var/lib/snapd/snaps/core_8268.snap)
/dev/loop3: [2304]:7093995 (/var/lib/snapd/snaps/lxd_12631.snap)Ok, so:
/dev/loop2: [2304]:7094294 (/var/snap/lxd/common/lxd/disks/default.img (deleted))
Confims that /var/snap/lxd/common got deleted somehow. It doesn’t appear to be the result of a snap deletion at least based on the history.
The good news is that we should be able to get all the container data out at least.
For that, try: mount /dev/loop2 /mnt
That should make the storage pool data show up in /mnt. If it does, back that up immediately, if your system reboots for any reason, it would all be gone.
Ok thanks for the help. All files show up now and i’m backing up. I’ll create the containers all over again.