Container changed to read-only file system

LXD
1

Out of the blue (as far as I can tell) one of my containers stopped allowing snapshots. Here is the error message:

$ lxc snapshot idea current
Error: Create instance snapshot: Failed to run: btrfs subvolume snapshot /var/snap/lxd/common/lxd/storage-pools/default/containers/idea /var/snap/lxd/common/lxd/storage-pools/default/containers-snapshots/idea/current: exit status 1 (ERROR: cannot snapshot '/var/snap/lxd/common/lxd/storage-pools/default/containers/idea': Read-only file system)

I have looked at dmesg (although I’m not sure what I’m looking for) and at some of the logs, but I can’t figure out what happened or how to correct this.

Thanks for any help.

2

I also notice this within the container. Doesn’t look good, but I don’t understand its implications:

# mount | grep snapfuse
snapfuse on /snap/core18/2654 type fuse.snapfuse (ro,nodev,relatime,user_id=0,group_id=0,allow_other)
snapfuse on /snap/core18/2667 type fuse.snapfuse (ro,nodev,relatime,user_id=0,group_id=0,allow_other)
snapfuse on /snap/core20/1738 type fuse.snapfuse (ro,nodev,relatime,user_id=0,group_id=0,allow_other)
snapfuse on /snap/core20/1778 type fuse.snapfuse (ro,nodev,relatime,user_id=0,group_id=0,allow_other)
snapfuse on /snap/lxd/21750 type fuse.snapfuse (ro,nodev,relatime,user_id=0,group_id=0,allow_other)
snapfuse on /snap/lxd/24164 type fuse.snapfuse (ro,nodev,relatime,user_id=0,group_id=0,allow_other)
snapfuse on /snap/lxd/24175 type fuse.snapfuse (ro,nodev,relatime,user_id=0,group_id=0,allow_other)
snapfuse on /snap/snapd/17883 type fuse.snapfuse (ro,nodev,relatime,user_id=0,group_id=0,allow_other)
snapfuse on /snap/snapd/17950 type fuse.snapfuse (ro,nodev,relatime,user_id=0,group_id=0,allow_other)
snapfuse on /snap/core18/2654 type fuse.snapfuse (ro,nodev,relatime,user_id=0,group_id=0,allow_other)
snapfuse on /snap/core18/2667 type fuse.snapfuse (ro,nodev,relatime,user_id=0,group_id=0,allow_other)
snapfuse on /snap/core20/1738 type fuse.snapfuse (ro,nodev,relatime,user_id=0,group_id=0,allow_other)
snapfuse on /snap/core20/1778 type fuse.snapfuse (ro,nodev,relatime,user_id=0,group_id=0,allow_other)
snapfuse on /snap/lxd/21750 type fuse.snapfuse (ro,nodev,relatime,user_id=0,group_id=0,allow_other)
snapfuse on /snap/lxd/24164 type fuse.snapfuse (ro,nodev,relatime,user_id=0,group_id=0,allow_other)
snapfuse on /snap/snapd/17883 type fuse.snapfuse (ro,nodev,relatime,user_id=0,group_id=0,allow_other)
snapfuse on /snap/snapd/17950 type fuse.snapfuse (ro,nodev,relatime,user_id=0,group_id=0,allow_other)
snapfuse on /snap/lxd/24239 type fuse.snapfuse (ro,nodev,relatime,user_id=0,group_id=0,allow_other)
snapfuse on /snap/core18/2654 type fuse.snapfuse (ro,nodev,relatime,user_id=0,group_id=0,allow_other)
snapfuse on /snap/core18/2667 type fuse.snapfuse (ro,nodev,relatime,user_id=0,group_id=0,allow_other)
snapfuse on /snap/core20/1738 type fuse.snapfuse (ro,nodev,relatime,user_id=0,group_id=0,allow_other)
snapfuse on /snap/core20/1778 type fuse.snapfuse (ro,nodev,relatime,user_id=0,group_id=0,allow_other)
snapfuse on /snap/lxd/21750 type fuse.snapfuse (ro,nodev,relatime,user_id=0,group_id=0,allow_other)
snapfuse on /snap/lxd/24164 type fuse.snapfuse (ro,nodev,relatime,user_id=0,group_id=0,allow_other)
snapfuse on /snap/snapd/17883 type fuse.snapfuse (ro,nodev,relatime,user_id=0,group_id=0,allow_other)
snapfuse on /snap/snapd/17950 type fuse.snapfuse (ro,nodev,relatime,user_id=0,group_id=0,allow_other)
snapfuse on /snap/lxd/24323 type fuse.snapfuse (ro,nodev,relatime,user_id=0,group_id=0,allow_other)
snapfuse on /snap/core18/2667 type fuse.snapfuse (ro,nodev,relatime,user_id=0,group_id=0,allow_other)
snapfuse on /snap/core20/1778 type fuse.snapfuse (ro,nodev,relatime,user_id=0,group_id=0,allow_other)
snapfuse on /snap/lxd/21750 type fuse.snapfuse (ro,nodev,relatime,user_id=0,group_id=0,allow_other)
snapfuse on /snap/lxd/24164 type fuse.snapfuse (ro,nodev,relatime,user_id=0,group_id=0,allow_other)
snapfuse on /snap/snapd/17950 type fuse.snapfuse (ro,nodev,relatime,user_id=0,group_id=0,allow_other)
snapfuse on /snap/core18/2697 type fuse.snapfuse (ro,nodev,relatime,user_id=0,group_id=0,allow_other)
snapfuse on /snap/lxd/24483 type fuse.snapfuse (ro,nodev,relatime,user_id=0,group_id=0,allow_other)
snapfuse on /snap/core18/2667 type fuse.snapfuse (ro,nodev,relatime,user_id=0,group_id=0,allow_other)
snapfuse on /snap/core20/1778 type fuse.snapfuse (ro,nodev,relatime,user_id=0,group_id=0,allow_other)
snapfuse on /snap/lxd/21750 type fuse.snapfuse (ro,nodev,relatime,user_id=0,group_id=0,allow_other)
snapfuse on /snap/lxd/24164 type fuse.snapfuse (ro,nodev,relatime,user_id=0,group_id=0,allow_other)
snapfuse on /snap/snapd/17950 type fuse.snapfuse (ro,nodev,relatime,user_id=0,group_id=0,allow_other)
snapfuse on /snap/snapd/18357 type fuse.snapfuse (ro,nodev,relatime,user_id=0,group_id=0,allow_other)
3

Looks similar to Error: mkdir /var/snap/lxd/common/lxd/shmounts/test: read-only file system when launching a container

4

Thanks for responding. I agree, it looks similar, but I don’t understand the system well enough to troubleshoot adequately. For example, I don’t understand the use or importance that shmounts has in the LXD environment. My directory /var/snap/lxd/common/shmounts on the LXD host is empty. Is that normal?

Regardless, I really have no idea how to troubleshoot this. I also don’t know how to change the environment to rw.

5

Could someone explain how I change a btrfs subvolume (I think that’s what I need to change) from ro to rw in the context of LXD? Thanks.

6

@johnmaher You may have ran out of space/quota? according to the docs the default pool size is “20% of free disk space, >= 5 GiB and <= 30 GiB”

7

Thank you @k4my4b. Wouldn’t that effect all containers?

8

@johnmaher not really, each container has its own volume (maybe on the same storage pool but different volumes)

9

Thank you. I’ll look into that.

10

There seems to be plenty of space, although admittedly I don’t know exactly how to determine this. I do get this:

$ sudo btrfs filesystem usage /var/snap/lxd/common/lxd/storage-pools/default/containers/idea
Overall:
    Device size:		   1.73TiB
    Device allocated:		 850.02GiB
    Device unallocated:		 924.12GiB
    Device missing:		     0.00B
    Used:			 493.97GiB
    Free (estimated):		   1.20TiB	(min: 766.57GiB)
    Free (statfs, df):		   1.20TiB
    Data ratio:			      1.00
    Metadata ratio:		      2.00
    Global reserve:		 512.00MiB	(used: 0.00B)
    Multiple profiles:		        no

Data,single: Size:770.01GiB, Used:465.50GiB (60.45%)
   /dev/md2p1	 770.01GiB

Metadata,DUP: Size:40.00GiB, Used:14.23GiB (35.58%)
   /dev/md2p1	  80.00GiB

System,DUP: Size:8.00MiB, Used:112.00KiB (1.37%)
   /dev/md2p1	  16.00MiB

Unallocated:
   /dev/md2p1	 924.12GiB

Seems like there are no space problems?

11

And here’s an interesting twist. I can create snapshots. Just not snapshots named “current”, which I had previously created and deleted (daily) just fine.

The output of:

$ lxc snapshot idea current
Error: Create instance snapshot: Failed to run: btrfs subvolume snapshot /var/snap/lxd/common/lxd/storage-pools/default/containers/idea /var/snap/lxd/common/lxd/storage-pools/default/containers-snapshots/idea/current: exit status 1 (ERROR: cannot snapshot '/var/snap/lxd/common/lxd/storage-pools/default/containers/idea': Read-only file system)

seeems to be only related to the snapshot called “current”, which doesn’t exist (isn’t listed) based on lxc info idea.

I still don’t know how to convert to read-write the file system it’s complaining about.

12

@johnmaher interesting! maybe @tomp can shed some light on this.

13

That would be most welcome.

14

So, I’m still stuck on this. I did notice that running lxc info idea shows no snapshot called “current”, but ls -l /var/snap/lxd/common/lxd/storage-pools/default/containers-snapshots/idea shows a directory called “current”.

Anyone know how to rectify this issue?

Thanks.

15

@johnmaher
I do not know if lxd creates subvolumes for each snapshot or if they’re just directories but you can backup current then delete it or just rename it to .e.g current.backup and try again

16

@k4my4b, might that create a conflict with the underlying database if I just delete the directory /var/snap/lxd/common/lxd/storage-pools/default/containers-snapshots/idea/current ?

I truly don’t know.

17

Although I just ran this:

$ lxd sql global 'select storage_volumes.name, storage_volumes_snapshots.* from storage_volumes_snapshots join storage_volumes on storage_volumes_snapshots.storage_volume_id = storage_volumes.id' | grep "idea"
| idea             | 2057 | 1151              | pre-modify-token-description-page |             | 0001-01-01T00:00:00Z |
| idea             | 3210 | 1151              | now                               |             | 0001-01-01T00:00:00Z |
| idea             | 3286 | 1151              | now2                              |             | 0001-01-01T00:00:00Z |
| idea             | 3287 | 1151              | now3                              |             | 0001-01-01T00:00:00Z |
| idea             | 3288 | 1151              | now4                              |             | 0001-01-01T00:00:00Z |
| idea             | 3481 | 1151              | now5                              |             | 0001-01-01T00:00:00Z |

which leads me to believe (but not necessarily know) that the database does not contain a reference to a snapshot called “current” in the “idea” instance.

18

@johnmaher
checks out
instead of deleting it straight away just rename it and see

19

Shazam! It worked. Thanks @k4my4b.

1 Like