LVM storage issue while starting container after core20 switch

[sfx]

Hello,

My container failed to start this morning with the following error:

Error: Failed preparing container for start: Failed to activate LVM logical volume "/dev/default/containers_ubuntu--20--10": Failed to run: lvchange --activate y --ignoreactivationskip /dev/default/containers_ubuntu--20--10: Volume group "default" not found
      Cannot process volume group default
    Try `lxc info --show-log ubuntu-20-10` for more info

I checked this previous thread : lvm-storage-problem-all-container-stopped-and-doesnt-start-anymore-after-switch-to-core20, but I am not using nested LVM and the snap set lxd lvm.external=true ; systemctl reload snap.lxd.daemon did not solve my problem.

$ lxc storage ls
+---------+--------+---------+-------------+---------+
|  NAME   | DRIVER | SOURCE  | DESCRIPTION | USED BY |
+---------+--------+---------+-------------+---------+
| default | lvm    | default |             | 4       |
+---------+--------+---------+-------------+---------+

$ lxc storage show default
config:
  lvm.thinpool_name: LXDThinPool
  lvm.vg_name: default
  source: default
  volatile.initial_source: /dev/mapper/nvme-lxd
description: ""
name: default
driver: lvm
used_by:
- /1.0/images/6878c599506131bb930941e56074b3718e8e22a250e92c61aae82e18c20c087a
- /1.0/instances/popular-firefly
- /1.0/instances/ubuntu-20-10
- /1.0/profiles/default
status: Created
locations:
- none

$ sudo vgs
  WARNING: PV /dev/sdb in VG vg0 is using an old PV header, modify the VG to update.
  WARNING: PV /dev/sdc in VG vg0 is using an old PV header, modify the VG to update.
  VG                       #PV #LV #SN Attr   VSize    VFree   
  nvme                       1   6   0 wz--n- <931,01g <337,59g
  renndlxrdl3786-bcache-vg   1   2   0 wz--n-   <7,28t       0 
  vg0                        2   1   0 wz--n-   <7,28t       0 

$ sudo lvs -a
  WARNING: PV /dev/sdb in VG vg0 is using an old PV header, modify the VG to update.
  WARNING: PV /dev/sdc in VG vg0 is using an old PV header, modify the VG to update.
  LV        VG                       Attr       LSize   Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert
  archlinux nvme                     -wi-a----- 207,75g                                                    
  cache     nvme                     -wi-ao----  96,00g                                                    
  cache2    nvme                     -wi-ao----  30,00g                                                    
  lxd       nvme                     -wi-a-----  50,00g                                                    
  swap      nvme                     -wi-ao----   4,00g                                                    
  ubuntu    nvme                     -wi-ao---- 205,67g                                                    
  local     renndlxrdl3786-bcache-vg -wi-ao----   7,26t                                                    
  swap      renndlxrdl3786-bcache-vg -wi-a-----  16,00g                                                    
  storage   vg0                      -wi-ao----  <7,28t 

I am not quite familiar with thin pools, from what I understand I would have expected the lvs command above to display thin pool information, especially LXDThinPool?

Thank you for your insight.

[tomp (Thomas Parrott)]

Its likely you’ve been affected by the snap core20 switch.

Does your /dev/mapper/nvme-lxd exist on another LVM volume?

Please see:

LXD snap transitioning to core20 and losing i386 support

There have been a few issues caused by the switch to core20 which I thought would be worth documenting in case any body is affected: Core20 prevented LXD’s ability to detect and use the xtables firewall driver LXD supports two firewall drivers; xtables (iptables, ip6tables, ebtables and arptables), and nftables. LXD will detect the presence of the firewall commands available, as well as the rule sets already in use, and will pick what it thinks is the most appropriate firewall driver to use. …

and

LVM storage problem - all container stopped and doesn't start anymore after switch to core20

Hello, I have a huge problem: all containers stopped and doesn’t start anymore. When I try to start a container I have this error: Error: Failed preparing container for start: Failed to activate LVM logical volume "/dev/pool_0/containers_iceCast": Failed to run: lvchange --activate y --ignoreactivationskip /dev/pool_0/containers_iceCast: Volume group "pool_0" not found Cannot process volume group pool_0 But with vgs/lvs commands I see vg and lv volumes. Some one have experienced this kind …

[tomp (Thomas Parrott)]

Ah I see using external tooling doesn’t work for you.

What is your host OS version?

[tomp (Thomas Parrott)]

Please can you show the output of the following commands on the LXD host:

sudo lvs
sudo vgs
sudo pvs

Then also from inside the snap using:

sudo LD_LIBRARY_PATH=/snap/lxd/current/lib/:/snap/lxd/current/lib/x86_64-linux-gnu/ nsenter --mount=/run/snapd/ns/lxd.mnt -- /snap/lxd/current/bin/lvs
sudo LD_LIBRARY_PATH=/snap/lxd/current/lib/:/snap/lxd/current/lib/x86_64-linux-gnu/ nsenter --mount=/run/snapd/ns/lxd.mnt -- /snap/lxd/current/bin/vgs
sudo LD_LIBRARY_PATH=/snap/lxd/current/lib/:/snap/lxd/current/lib/x86_64-linux-gnu/ nsenter --mount=/run/snapd/ns/lxd.mnt -- /snap/lxd/current/bin/pvs

[sfx]

Host OS version:

$ lsb_release -a
No LSB modules are available.
Distributor ID:	Ubuntu
Description:	Ubuntu 20.04.2 LTS
Release:	20.04
Codename:	focal

Host:

$ sudo lvs
  WARNING: PV /dev/sdb in VG vg0 is using an old PV header, modify the VG to update.
  WARNING: PV /dev/sdc in VG vg0 is using an old PV header, modify the VG to update.
  LV        VG                       Attr       LSize   Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert
  archlinux nvme                     -wi-a----- 207,75g                                                    
  cache     nvme                     -wi-ao----  96,00g                                                    
  cache2    nvme                     -wi-ao----  30,00g                                                    
  lxd       nvme                     -wi-a-----  50,00g                                                    
  swap      nvme                     -wi-ao----   4,00g                                                    
  ubuntu    nvme                     -wi-ao---- 205,67g                                                    
  local     renndlxrdl3786-bcache-vg -wi-ao----   7,26t                                                    
  swap      renndlxrdl3786-bcache-vg -wi-a-----  16,00g                                                    
  storage   vg0                      -wi-ao----  <7,28t                                                    

$ sudo vgs
  WARNING: PV /dev/sdb in VG vg0 is using an old PV header, modify the VG to update.
  WARNING: PV /dev/sdc in VG vg0 is using an old PV header, modify the VG to update.
  VG                       #PV #LV #SN Attr   VSize    VFree   
  nvme                       1   6   0 wz--n- <931,01g <337,59g
  renndlxrdl3786-bcache-vg   1   2   0 wz--n-   <7,28t       0 
  vg0                        2   1   0 wz--n-   <7,28t       0 

$ sudo pvs
  WARNING: PV /dev/sdb in VG vg0 is using an old PV header, modify the VG to update.
  WARNING: PV /dev/sdc in VG vg0 is using an old PV header, modify the VG to update.
  PV             VG                       Fmt  Attr PSize    PFree   
  /dev/bcache0   renndlxrdl3786-bcache-vg lvm2 a--    <7,28t       0 
  /dev/nvme0n1p2 nvme                     lvm2 a--  <931,01g <337,59g
  /dev/sdb       vg0                      lvm2 a--    <3,64t       0 
  /dev/sdc       vg0                      lvm2 a--    <3,64t       0

Inside snap:

$ sudo nsenter --mount=/run/snapd/ns/lxd.mnt -- bash
/# LD_LIBRARY_PATH=/snap/lxd/current/lib/:/snap/lxd/current/lib/x86_64-linux-gnu/ /snap/lxd/current/bin/lvs
  WARNING: PV /dev/sdb in VG vg0 is using an old PV header, modify the VG to update.
  WARNING: PV /dev/sdc in VG vg0 is using an old PV header, modify the VG to update.
  LV        VG                       Attr       LSize   Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert
  archlinux nvme                     -wi-a----- 207.75g                                                    
  cache     nvme                     -wi-ao----  96.00g                                                    
  cache2    nvme                     -wi-ao----  30.00g                                                    
  lxd       nvme                     -wi-a-----  50.00g                                                    
  swap      nvme                     -wi-ao----   4.00g                                                    
  ubuntu    nvme                     -wi-ao---- 205.67g                                                    
  local     renndlxrdl3786-bcache-vg -wi-ao----   7.26t                                                    
  swap      renndlxrdl3786-bcache-vg -wi-a-----  16.00g                                                    
  storage   vg0                      -wi-ao----  <7.28t                                                    

/# LD_LIBRARY_PATH=/snap/lxd/current/lib/:/snap/lxd/current/lib/x86_64-linux-gnu/ /snap/lxd/current/bin/vgs
  WARNING: PV /dev/sdb in VG vg0 is using an old PV header, modify the VG to update.
  WARNING: PV /dev/sdc in VG vg0 is using an old PV header, modify the VG to update.
  VG                       #PV #LV #SN Attr   VSize    VFree   
  nvme                       1   6   0 wz--n- <931.01g <337.59g
  renndlxrdl3786-bcache-vg   1   2   0 wz--n-   <7.28t       0 
  vg0                        2   1   0 wz--n-   <7.28t       0 

/# LD_LIBRARY_PATH=/snap/lxd/current/lib/:/snap/lxd/current/lib/x86_64-linux-gnu/ /snap/lxd/current/bin/pvs
  WARNING: PV /dev/sdb in VG vg0 is using an old PV header, modify the VG to update.
  WARNING: PV /dev/sdc in VG vg0 is using an old PV header, modify the VG to update.
  PV             VG                       Fmt  Attr PSize    PFree   
  /dev/bcache0   renndlxrdl3786-bcache-vg lvm2 a--    <7.28t       0 
  /dev/nvme0n1p2 nvme                     lvm2 a--  <931.01g <337.59g
  /dev/sdb       vg0                      lvm2 a--    <3.64t       0 
  /dev/sdc       vg0                      lvm2 a--    <3.64t       0 

[tomp (Thomas Parrott)]

Right so I suspect that because you’re running 20.04 on the host, you’re host side tools are now having the same issue as the tools inside the snap (which are also using 20.04, aka core20).

I suspect if you modify /etc/lvm/lvm.conf and find scan_lvs = 0 and set it to scan_lvs = 1 and then restart your system, and keep snap set lxd lvm.external=true then you should see the default volume group in the output from your host side sudo vgs command.

I suspect you were actually inadvertently depending on the old snap core18 behaviour of always scanning nested logical volumes.

[sfx]

Eveything works fine after following your instructions!
Thanks a lot for your (quick) support.

$ sudo lvs
  WARNING: PV /dev/sdb in VG vg0 is using an old PV header, modify the VG to update.
  WARNING: PV /dev/sdc in VG vg0 is using an old PV header, modify the VG to update.
  LV                                                                      VG                       Attr       LSize   Pool        Origin Data%  Meta%  Move Log Cpy%Sync Convert
  LXDThinPool                                                             default                  twi-aotz-- <48,00g                    17,15  1,96                            
  containers_popular--firefly                                             default                  Vwi---tz-k  <9,32g LXDThinPool                                               
  containers_ubuntu--20--10                                               default                  Vwi---tz-k  <9,32g LXDThinPool                                               
  images_6878c599506131bb930941e56074b3718e8e22a250e92c61aae82e18c20c087a default                  Vwi---tz-k  <9,32g LXDThinPool                                               
  archlinux                                                               nvme                     -wi-a----- 207,75g                                                           
  cache                                                                   nvme                     -wi-ao----  96,00g                                                           
  cache2                                                                  nvme                     -wi-ao----  30,00g                                                           
  lxd                                                                     nvme                     -wi-ao----  50,00g                                                           
  swap                                                                    nvme                     -wi-ao----   4,00g                                                           
  ubuntu                                                                  nvme                     -wi-ao---- 205,67g                                                           
  local                                                                   renndlxrdl3786-bcache-vg -wi-ao----   7,26t                                                           
  swap                                                                    renndlxrdl3786-bcache-vg -wi-a-----  16,00g                                                           
  storage                                                                 vg0                      -wi-ao----  <7,28t 

[tomp (Thomas Parrott)]

Excellent glad it sorted it.