「LXC 4.23」unable to delete lxd virtual machine snapshot

LXD
1

Hello all,

I find out that my lxd server can launch/start/stop/snapshot the virtual machine, but when I try to delete the snapshot, it throws a error:

oot@dev-physical-0-19:~# lxc list 
+------------------------------+---------+---------------------------------+------+-----------------+-----------+
|             NAME             |  STATE  |              IPV4               | IPV6 |      TYPE       | SNAPSHOTS |
+------------------------------+---------+---------------------------------+------+-----------------+-----------+
| centos-7-base                | STOPPED |                                 |      | CONTAINER       | 1         |
+------------------------------+---------+---------------------------------+------+-----------------+-----------+
| centos-7-kvm-base            | STOPPED |                                 |      | VIRTUAL-MACHINE | 1         |
+------------------------------+---------+---------------------------------+------+-----------------+-----------+
| centos-8-base                | STOPPED |                                 |      | CONTAINER       | 1         |
+------------------------------+---------+---------------------------------+------+-----------------+-----------+
| centos-8-kvm-base            | STOPPED |                                 |      | VIRTUAL-MACHINE | 2         |

root@dev-physical-0-19:~# ll /data/lxc/virtual-machines-snapshots/centos-8-kvm-base/
total 0
drwx------ 4 root root  40 Mar  9 10:25 ./
drwx--x--x 4 root root  68 Mar  9 03:39 ../
d--x------ 4 root root 237 Mar  8 05:19 1.0/
d--x------ 4 root root 237 Mar  8 05:19 2.0/

root@dev-physical-0-19:~# lxc delete centos-8-kvm-base/1.0
Error: Failed to remove '/var/snap/lxd/common/lxd/storage-pools/lxc-storage/virtual-machines-snapshots/centos-8-kvm-base/1.0': unlinkat /var/snap/lxd/common/lxd/storage-pools/lxc-storage/virtual-machines-snapshots/centos-8-kvm-base/1.0/config: read-only file system

root@dev-physical-0-19:~# lxc delete centos-8-kvm-base/2.0
Error: Failed to remove '/var/snap/lxd/common/lxd/storage-pools/lxc-storage/virtual-machines-snapshots/centos-8-kvm-base/2.0': unlinkat /var/snap/lxd/common/lxd/storage-pools/lxc-storage/virtual-machines-snapshots/centos-8-kvm-base/2.0/config: read-only file system

I try to solve the problem, but the directory is very strange:

root@dev-physical-0-19:~# ll /var/snap/lxd/common/lxd/storage-pools/lxc-storage/virtual-machines-snapshots/
ls: cannot access '/var/snap/lxd/common/lxd/storage-pools/lxc-storage/virtual-machines-snapshots/': No such file or directory
root@dev-physical-0-19:~# ll /var/snap/lxd/common/lxd/storage-pools/lxc-storage
total 8
drwx--x--x 2 root root 4096 Mar  8 04:25 ./
drwx--x--x 3 root root 4096 Mar  8 05:37 ../

The directory is not exist. I am failing to understand this issue and how to fix it.

Env Information:

root@dev-physical-0-19:~# lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 21.10
Release:        21.10
Codename:       impish

root@dev-physical-0-19:~# lxc version
Client version: 4.23
Server version: 4.23

root@dev-physical-0-19:~# snap services
Service          Startup  Current   Notes
lxd.activate     enabled  inactive  -
lxd.daemon       enabled  active    socket-activated
lxd.user-daemon  enabled  inactive  socket-activated

root@dev-physical-0-19:~# snap logs lxd
2022-03-08T07:50:06Z lxd.daemon[2738]: - loadavg_daemon
2022-03-08T07:50:06Z lxd.daemon[2738]: - pidfds
2022-03-08T07:50:07Z lxd.daemon[2477]: => Starting LXD
2022-03-08T07:50:07Z lxd.daemon[2757]: t=2022-03-08T07:50:07+0000 lvl=warn msg=" - Couldn't find the CGroup blkio.weight, disk priority will be ignored"
2022-03-08T07:50:08Z systemd[1]: snap.lxd.activate.service: Deactivated successfully.
2022-03-08T07:50:08Z systemd[1]: Finished Service for snap application lxd.activate.
2022-03-08T07:50:09Z lxd.daemon[2477]: => LXD is ready
2022-03-09T02:35:55Z lxd.daemon[2757]: t=2022-03-09T02:35:55+0000 lvl=warn msg="Detected poll(POLLNVAL) event."
2022-03-09T10:39:49Z lxd.daemon[2757]: t=2022-03-09T10:39:49+0000 lvl=eror msg="DeleteSnapshots: Failed to delete the snapshot" err="Failed to remove '/var/snap/lxd/common/lxd/storage-pools/lxc-storage/virtual-machines-snapshots/centos-8-kvm-base/1.0': unlinkat /var/snap/lxd/common/lxd/storage-pools/lxc-storage/virtual-machines-snapshots/centos-8-kvm-base/1.0/config: read-only file system" instance=centos-8-kvm-base project=default snapshot=centos-8-kvm-base/1.0
2022-03-09T10:39:49Z lxd.daemon[2757]: t=2022-03-09T10:39:49+0000 lvl=eror msg="DeleteSnapshots: Failed to delete the snapshot" err="Failed to remove '/var/snap/lxd/common/lxd/storage-pools/lxc-storage/virtual-machines-snapshots/centos-8-kvm-base/2.0': unlinkat /var/snap/lxd/common/lxd/storage-pools/lxc-storage/virtual-machines-snapshots/centos-8-kvm-base/2.0/config: read-only file system" instance=centos-8-kvm-base project=default snapshot=centos-8-kvm-base/2.0
2

I have make some progress with the problem, I just restart the snapd service, but it works:

root@dev-physical-0-19:~# snap restart lxd 
Restarted.

root@dev-physical-0-19:~# snap  services 
Service          Startup  Current   Notes
lxd.activate     enabled  inactive  -
lxd.daemon       enabled  active    socket-activated
lxd.user-daemon  enabled  inactive  socket-activated

root@dev-physical-0-19:~# lxc delete centos-8-kvm-base/1.0
root@dev-physical-0-19:~# ll /data/lxc/virtual-machines-snapshots/centos-8-kvm-base/
total 0
drwx------ 3 root root  25 Mar  9 10:59 ./
drwx--x--x 4 root root  68 Mar  9 03:39 ../
d--x------ 4 root root 237 Mar  8 05:19 2.0/

I think there must be something wrong with the snapd that controlling the lxd services.

3

Can you show output of lxc storage show lxc-storage please?

4

I just simply use directory mode:

root@dev-physical-0-19:~# lxc storage list 
+-------------+--------+------------+-------------+---------+
|    NAME     | DRIVER |   SOURCE   | DESCRIPTION | USED BY |
+-------------+--------+------------+-------------+---------+
| lxc-storage | dir    | /data/lxc/ |             | 11      |

+-------------+--------+------------+-------------+---------+
root@dev-physical-0-19:~# lxc storage show lxc-storage
config:
  source: /data/lxc/
description: ""
name: lxc-storage
driver: dir
used_by:
- /1.0/instances/centos-7-base
- /1.0/instances/centos-7-kvm-base
- /1.0/instances/centos-8-base
- /1.0/instances/centos-8-kvm-base
- /1.0/instances/dev-host-0-31-jumper
- /1.0/instances/dev-host-0-36-qa-enteprise-3
- /1.0/instances/dev-host-0-39-qa-enteprise-5
- /1.0/instances/dev-host-0-44-qa-enteprise-7
- /1.0/instances/dev-host-0-47-qa-enteprise-7
- /1.0/instances/dev-host-0-50-dns
- /1.0/profiles/default
status: Created
locations:
- none
5

So restarting lxd service allowed you to delete the snapshot?

6

Yes, and I also found out some relevant appearance:

Before I restart the snapd service, the directory 「/var/snap/lxd/common/shmounts/」 is unavailable link.

But after that, now it’s just a normal directory:

root@dev-physical-0-19:~# ll /var/snap/lxd/common/
total 48
drwxr-xr-x  9 root root 4096 Mar  9 10:58 ./
drwxr-xr-x  5 root root 4096 Mar  8 04:24 ../
-rw-r--r--  1 root root  324 Mar  8 04:24 config
drwxr-xr-x  2 root root 4096 Mar  8 03:56 global-conf/
drwxr-xr-x  2 root root 4096 Mar  8 03:56 lxc/
-rw-------  1 root root    7 Mar  9 10:58 lxcfs.pid
drwx--x--x 17 root root 4096 Mar  9 10:58 lxd/
-rw-r--r--  1 root root    7 Mar  9 10:58 lxd.pid
drwxr-xr-x  2 root root 4096 Mar  8 07:49 lxd-user/
lrwxrwxrwx  1 root root   17 Mar  9 10:58 mntns -> /proc/429762/root/
drwx------  2 root root 4096 Mar  8 03:56 ns/
drwx--x--x  2 root root 4096 Mar  8 03:56 shmounts/
-rw-r--r--  1 root root    0 Mar  9 10:58 state
drwxr-xr-x  3 root root 4096 Mar  8 03:56 var/