Lxc commands fails with socket error on LXD 3.14

hatzlhoffer · June 30, 2019, 1:30am

ls -lh /var/snap/lxd/common/mntns/var/snap/lxd/common/lxd/snapshots

total 12K
lrwxrwxrwx 1 root root 76 Nov  8  2018 dbserver -> /var/snap/lxd/common/lxd/storage-pools/default/containers-snapshots/dbserver
lrwxrwxrwx 1 root root 78 Nov  8  2018 mailserver -> /var/snap/lxd/common/lxd/storage-pools/default/containers-snapshots/mailserver
lrwxrwxrwx 1 root root 77 Jun 30 02:59 webserver -> /var/snap/lxd/common/lxd/storage-pools/default/containers-snapshots/webserver

ls -lh /var/snap/lxd/common/mntns/var/snap/lxd/common/lxd/snapshots/webserver

lrwxrwxrwx 1 root root 77 Jun 30 02:59 /var/snap/lxd/common/mntns/var/snap/lxd/common/lxd/snapshots/webserver -> /var/snap/lxd/common/lxd/storage-pools/default/containers-snapshots/webserver

ls -lh /var/snap/lxd/common/mntns/var/snap/lxd/common/lxd/storage-pool/default/containers-snapshots

ls: cannot access '/var/snap/lxd/common/mntns/var/snap/lxd/common/lxd/storage-pool/default/containers-snapshots': No such file or directory

ls -lh /var/snap/lxd/common/mntns/var/snap/lxd/common/lxd/storage-pool/default/containers-snapshots/webserver

ls: cannot access '/var/snap/lxd/common/mntns/var/snap/lxd/common/lxd/storage-pool/default/containers-snapshots/webserver': No such file or directory

hatzlhoffer · June 30, 2019, 1:42am

ok…there seems to be a typo:

root@indianer2:/home/michael# ls -lh /var/snap/lxd/common/mntns/var/snap/lxd/common/lxd/storage-pools/default/containers-snapshots/webserver/

total 0
drwxr-xr-x+ 1 100000 100000 78 Sep 26  2018 20181108
drwxr-xr-x+ 1 100000 100000 78 Sep 26  2018 snap3
drwxr-xr-x+ 1 100000 100000 78 Sep 26  2018 snap4
drwxr-xr-x+ 1 100000 100000 78 Sep 26  2018 snap5
drwx--x--x+ 1 root   root   78 Sep 26  2018 snap6

root@indianer2:/home/michael# ls -lh /var/snap/lxd/common/mntns/var/snap/lxd/common/lxd/storage-pools/default/containers-snapshots

total 0
drwx--x--x 1 root root 76 Jun 30 02:37 dbserver
drwx--x--x 1 root root 86 Mai 13 10:49 mailserver
drwx--x--x 1 root root 76 Mai 13 10:50 mailtools
drwx--x--x 1 root root 56 Jun 30 02:38 webserver

stgraber · June 30, 2019, 2:19am

Ok, can you see if:

sudo rm -Rf /var/snap/lxd/common/mntns/var/snap/lxd/common/lxd/storage-pools/default/containers/webserver
sudo mv /var/snap/lxd/common/mntns/var/snap/lxd/common/lxd/storage-pools/default/containers-snapshots/webserver/snap5 /var/snap/lxd/common/mntns/var/snap/lxd/common/lxd/storage-pools/default/containers/webserver
lxc start webserver

Gets things back online somehow?

hatzlhoffer · June 30, 2019, 2:37am

man…

…you made it!!!
my webserver is online again!

hatzlhoffer · June 30, 2019, 2:38am

wohoooo!

Thank you very much!

Can you say what the problem was?

hatzlhoffer · June 30, 2019, 2:42am

ok…just one thing:

i tried to start an export to be sure:

michael@indianer2:~/backup/container/webserver$ lxc export webserver webserver_20190630.tar.gz
Error: Create backup: Backup storage: subvolume snapshot failed, source=/var/snap/lxd/common/lxd/storage-pools/default/containers/webserver, dest=/var/snap/lxd/common/lxd/storage-pools/default/containers/webserver055383746/.backup, output=ERROR: not a subvolume: /var/snap/lxd/common/lxd/storage-pools/default/containers/webserver

this doesnt work.
would this be a further problem?

hatzlhoffer · June 30, 2019, 2:46am

for the other containers also didnt work:

michael@indianer2:~/backup/container/mailserver$ lxc export mailserver mailserver_20190630.tar.gz
Error: Create backup: Backup storage: subvolume snapshot failed, source=/var/snap/lxd/common/lxd/storage-pools/default/containers-snapshots/mailserver/20181108.ro, dest=/var/snap/lxd/common/lxd/storage-pools/default/containers-snapshots/mailserver/20181108, output=ERROR: not a subvolume: /var/snap/lxd/common/lxd/storage-pools/default/containers-snapshots/mailserver/20181108.ro

michael@indianer2:~/backup/container/mailserver$ cd ../mailtools
michael@indianer2:~/backup/container/mailtools$ lxc export mailtools mailtools_20190630.tar.gz
Error: Create backup: Backup storage: subvolume snapshot failed, source=/var/snap/lxd/common/lxd/storage-pools/default/containers/mailtools, dest=/var/snap/lxd/common/lxd/storage-pools/default/containers/mailtools771957203/.backup, output=ERROR: not a subvolume: /var/snap/lxd/common/lxd/storage-pools/default/containers/mailtools

stgraber · June 30, 2019, 3:35am

Sounds like a lot of those containers aren’t on btrfs subvolumes, this can get fixed manually by creating btrfs subvolumes in those locations and moving the data over.

I have no idea how volumes on a btrfs storage pool aren’t using btrfs though.

Can you show `nsenter -t $(pgrep daemon.start) -m – /snap/lxd/current/bin/btrfs subvol show /var/snap/lxd/common/lxd/storage-pools/default/

hatzlhoffer · June 30, 2019, 10:52am

is it possible that there is a – too much in the command?
If i execute this i get

nsenter: failed to execute –: No such file or directory

if i execute:

nsenter -t $(pgrep daemon.start) -m /snap/lxd/current/bin/btrfs subvol show /var/snap/lxd/common/lxd/storage-pools/default/

i get:

/var/snap/lxd/common/lxd/storage-pools/default is toplevel subvolume

short question:
I dont need the snapshots anymore if i have my systems running and am able to export the containers.

Can i simply delete all the snapshots and may be able to export the containers then?
Or wouldn’t that fix the problem?

hatzlhoffer · June 30, 2019, 10:53am

other point:
i am absolutely thankful that you helped me so much!

stgraber · July 1, 2019, 9:58pm

If you don’t need the snapshots, I would recommend you atempt to delete them from LXD, then get rid of whatever’s left on the filesystem.

The output from the btrfs command above is a bit odd as far as the containers are concerned and this will give you trouble when making snapshots or exports down the line.

Once the snapshots are gone, the next step would to manually convert those container directories back into subvolumes so that btrfs subvol show gets you one entry per container, once that’s done, snapshots, exports, … will all work again properly.

hatzlhoffer · July 2, 2019, 9:01am

i deleted all snapshots via

lxc delete container/snapshot

now the output of

lxd sql global “SELECT * FROM containers;”

is

±—±--------±-----------±-------------±-----±----------±--------------------------±---------±------------------------------------±------------±-----------±---------------------+
| id | node_id | name | architecture | type | ephemeral | creation_date | stateful | last_use_date | description | project_id | expiry_date |
±—±--------±-----------±-------------±-----±----------±--------------------------±---------±------------------------------------±------------±-----------±---------------------+
| 1 | 1 | webserver | 2 | 0 | 0 | 2018-09-26T22:06:19+02:00 | 0 | 2019-07-02T10:45:55.608809434+02:00 | | 1 | 0001-01-01T00:00:00Z |
| 2 | 1 | mailserver | 2 | 0 | 0 | 2018-09-26T22:10:35+02:00 | 0 | 2019-07-02T10:45:52.053275161+02:00 | | 1 | 0001-01-01T00:00:00Z |
| 3 | 1 | mailtools | 2 | 0 | 0 | 2018-09-27T08:52:25+02:00 | 0 | 2019-07-02T10:45:54.099506826+02:00 | | 1 | 0001-01-01T00:00:00Z |
| 4 | 1 | dbserver | 2 | 0 | 0 | 2018-09-27T13:34:36+02:00 | 0 | 2019-07-02T10:45:50.486462266+02:00 | | 1 | 0001-01-01T00:00:00Z |
±—±--------±-----------±-------------±-----±----------±--------------------------±---------±------------------------------------±------------±-----------±---------------------+

what do i have to delete now? (and how)?

is it possible / will it work if i mount the default.img (e.g. to /mnt/def_leer) and then delete the content via

rm -Rf /mnt/def_leer/containers-snapshots/*

and

rm -Rf /mnt/def_leer/snapshots/*

as sudo?
Or will there be a problem with read-only file system (btrfs subvolumes)?

Edit: Added question for deleting the snapshots from the filesystem

stgraber · July 2, 2019, 5:26pm

You should definitely try those two as root.
If they don’t work, then each snapshot will likely need:

sudo btrfs property set PATH readonly no
sudo btrfs subvol delete PATH

hatzlhoffer · July 3, 2019, 11:28am

ok…both rm commands could be executed without problems as sudo.

can you tell me how i have to convert the container directories into subvolumes?

the command (as root)
nsenter -t $(pgrep daemon.start) -m /snap/lxd/current/bin/btrfs subvol show /var/snap/lxd/common/lxd/storage-pools/default/

still gives the output
/var/snap/lxd/common/lxd/storage-pools/default is toplevel subvolume

stgraber · July 4, 2019, 4:30am

Ok, so for that part you will need to stop all your containers, then mount the btrfs pool in /mnt.

Then for each directory in containers/ (one per container), you’ll need to:

mv NAME NAME.bak
btrfs subvolume create NAME
chown + chmod the newly created directory to match the old one
mv NAME.bak/* NAME/
rmdir NAME

After doing that for all containers, you’ll end up with a filesystem that looks identical in ls -lh but with each container now backed by a subvolume which can be confirmed with btrfs subvol show.

With that done, you should be able to snapshot containers just fine (as well as export, publish, …)

hatzlhoffer · July 4, 2019, 9:37am

Thank you, thank you, thank you!!!

Now everything works fine again. I was able to export all my containers again.
Snapshots can be made too.

just one little thing i changed:

i think this must be

rmdir NAME.bak

if i am not totally wrong - at least this worked for me.
Otherwise i would have deleted the files i already copied, i think.

Thanks a lot and best regards
Michael

stgraber · July 5, 2019, 3:30am

Oh, indeed, sorry about that
Great to hear that this all worked fine.