Error when trying to use an existing LVM pool with a new installation

LXD
1

I had to update my system from CentOS 7 to 8 which actually is a complete new installation including LXD. Until recently I could issue lxc storage create lvmpool lvm source=usrvg lvm.thinpool_name=lxd-thinpool to create a new storage pool from existing data. With version lxd 3.20 (the latest available on COPR) I get “Error: Volume group “usrvg” is not empty”. I get the same error if I omit lvm.thinpool_name to create a completely new LVM pool in an existing vg along with other data. In version 3.14 it used to work flawlessly that way.

Probably I miss something. Is there a new way to accomplish this?

Thanks

2

I believe this was a bug in prior versions of LXD.

Because LVM is a flat namespace, even when thinpools are used, we can’t allow LXD to use a VG that already has LVs as you may get name conflicts, causing LXD to potentially destroy your data.

@tomp

3

@stgraber yes that is correct, with or without thinpools, the logical volumes that LXD will create for instances and custom volumes appear in the same ‘namespace’ as any existing volumes in the volume group.

@res.digita can you show the output of lvs please.

Thanks
Tom

4

On the system I currently work on:

[root@gaia ~]# lvs 
  LV           VG    Attr       LSize   Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert
  root         sysvg -wi-ao----  20,00g                                                    
  swap         sysvg -wi-ao----   6,00g                                                    
  var_log      sysvg -wi-ao----   8,00g                                                    
  libvirt      usrvg -wi-ao----  20,00g                                                    
  lxd          usrvg -wi-ao----  10,00g                                                    
  lxdThinPool  usrvg twi-a-tz-- <99,90g             0,00   10,82                           
  ouranos-misc usrvg -wi-ao----  50,00g                                                    
  ouranos-sys  usrvg -wi-ao----  35,00g                                                    
  ouranos-usr  usrvg -wi-ao----  50,00g                                                    
  ourea_srv    usrvg -wi-ao---- 800,00g                                                    
  ourea_sys    usrvg -wi-ao----  35,00g                                                    
  pontos-misc  usrvg -wi-ao----  50,00g                                                    
  pontos-sys   usrvg -wi-ao----  35,00g                                                    
  pontos-usr   usrvg -wi-ao----  50,00g

LV lxd is mounted at /var/lib/lxd and lxdThinPool is meant to contain the containers and images (currently empty because of my experimentation).

On one of our other systems I find:

[root@hydra ~]# lvs 
  LV                                                                      VG     Attr       LSize  Pool         Origin Data%  Meta%  Move Log Cpy%Sync Convert
  images_475dfe1ed9d0a9935b19e31ae5e131037321f4a57b2b6f3069b768629b6dd069 fedora Vwi-a-tz-- 10,00g usr_thinpool        4,27                                   
  lv_cheiron                                                              fedora -wi-ao---- 54,00g                                                            
  root                                                                    fedora -wi-ao---- 15,00g                                                            
  sys_thinpool                                                            fedora twi-aotz--  5,00g                     9,60   13,62                           
  thin_var_lib_lxd                                                        fedora Vwi-aotz-- 10,00g sys_thinpool        4,80                                   
  usr_thinpool                                                            fedora twi-aotz-- 10,00g                     4,27   12,27                           
  var_lib_pgsql                                                           fedora -wi-ao---- 40,00g                                                            
  var_log                                                                 fedora -wi-ao----  5,00g

[root@hydra ~]# rpm -q lxd
lxd-3.16-0.1.fc30.x86_64

[root@hydra ~]# lxc storage list 
+---------+--------------+--------+--------+---------+
|  NAME   | BESCHREIBUNG | DRIVER | SOURCE | USED BY |
+---------+--------------+--------+--------+---------+
| lvmpool |              | lvm    | fedora | 2       |
+---------+--------------+--------+--------+---------+

[root@hydra ~]# lxc storage show lvmpool

config:
   lvm.thinpool_name: usr_thinpool
  lvm.vg_name: fedora
  source: fedora
  volatile.initial_source: fedora
  volume.block.filesystem: xfs
  description: ""
  name: lvmpool
  driver: lvm
  used_by:
  - /1.0/images/475dfe1ed9d0a9935b19e31ae5e131037321f4a57b2b6f3069b768629b6dd069. 
  - /1.0/profiles/default
 status: Created
 locations:
- none

As far as I understand it, LVM is intended to make the VGs a flexible container for a variety of different LVs. I think it is a bit disappointing when a storage pool requires a completely empty VG.

Specifically with CentoS / Fedora there is an issue with a VG containing just LXD thin pool. At boot time nothing is mounted and CentOS / Fedora used to ignore / don’t activate that VG at about 50% of reboots, unfortunately. Later, when LXD starts up it doesn’t find that VG.

Is there a workaround other than using a dir pool?

5

Yes so you can see that the images_... volume belongs to the usr_thinpool, but is in the same namespace as the other non-LXD volumes, so one could create LVs that conflict with LXD’s naming, or worse we might destroy an LV that isn’t actually owned by LXD. This was the reason for this check.

As for activating the volume group/thinpool, LXD should do this on startup, and doesn’t rely on the OS to do it (see https://github.com/lxc/lxd/blob/master/lxd/storage/drivers/driver_lvm.go#L498-L502), so if thats not happening something else is wrong.

6

Yes so you can see that the images_... volume belongs to the usr_thinpool , but is in the same namespace as the other non-LXD volumes, so one could create LVs that conflict with LXD’s naming, or worse we might destroy an LV that isn’t actually owned by LXD

Yes, it is the same namespace. But I bet LVM reliably prevents LVs with identical names from being created. So nothing should get accidentally overwritten or destroyed. But, sure, LXD needs some logic to handle a situation like this. Maybe adding a timestamp or a UUID to the name would be sufficient?

As for activating the volume group/thinpool, LXD should do this on startup, and doesn’t rely on the OS to do it (see https://github.com/lxc/lxd/blob/master/lxd/storage/drivers/driver_lvm.go#L498-L502), so if thats not happening something else is wrong.

OK, I made Fedora / Centos “responsible”, because there were no issues as soon as Fedora / CentOS created a /dev/lxdvg entry. I’ll test it with current versions and report here as well if the issue still exists.

7

I’ve the same issue with version 3.21 on Debian buster using snap. When I setup the server with version 3.18, the creation of storage pools using thin pools did work:

# lxc storage create pool-ssd lvm source=SSD lvm.thinpool_name=lxd-ssd-tp
Error: Volume group "SSD" is not empty

# lvs
  LV                                                                      VG  Attr       LSize    Pool       Origin Data%  Meta%  Move Log Cpy%Sync Convert
  containers_gitlabci--prod--rub1--01                                     HDD Vwi-aotz-- <195.58g lxd-hdd-tp        27.11                                  
  containers_pg11--intg--rub1--01                                         HDD Vwi-aotz--  372.53g lxd-hdd-tp        21.16                                  
  containers_samba--prod--rub1--01                                        HDD Vwi-aotz--  838.19g lxd-hdd-tp        1.71                                   
  containers_unifictrl--prod--rub1--01                                    HDD Vwi-aotz--   <9.32g lxd-hdd-tp        19.73                                  
  containers_wg--prod--rub1--01                                           HDD Vwi-aotz--   <9.32g lxd-hdd-tp        18.28                                  
  images_8d488209329b1480f4ea69320952085891555056bfb700c9199f962536e5a444 HDD Vwi-a-tz--   <9.32g lxd-hdd-tp        6.07                                   
  lxd-hdd-tp                                                              HDD twi-aotz--  640.00g                   23.48  20.38                           
  var-log                                                                 HDD -wi-ao----   18.62g                                                          
  lxd-ssd-tp                                                              SSD twi-a-tz--   40.00g                   0.00   10.48                           
  system                                                                  SSD -wi-ao----   18.62g                                                          
  var                                                                     SSD -wi-ao----   18.62g

According to my understanding of the documentation do the same as explained in the example:

* Use the existing LVM Thinpool called "my-pool" in Volume Group "my-vg".

lxc storage create pool1 lvm source=my-vg lvm.thinpool_name=my-pool
8

Yes that is the expected behaviour because the volume group is not empty. The intention for that check was to allow an existing (empty) volume group to be used to store instances, allowing for the user to create the volume group in more complex setups.

Being able to create a storage pool from a volume group that is non-empty was actually a bug in LXD until recently.

May I ask how you came to have a volume group with containers in it but a fresh installation of LXD? Have you recently reinstalled LXD?

9

I tried to determine what happens:

According to https://github.com/lxc/lxd/blob/lxd-3.21/lxd/storage_lvm.go the method lvmGetLVCount(poolName) is called. When I execute the system command in this method (vgs --noheadings -o lvcount SSD) I get 3 as result.

Later I assue the following code is executed:

if count > 0 && s.useThinpool {
	ok, err := storageLVMThinpoolExists(poolName, s.thinPoolName)
	if err != nil {
		logger.Errorf("Failed to determine whether thinpool \"%s\" exists in volume group \"%s\": %s", poolName, s.thinPoolName, err)
		return err
	}
	empty = ok
}

When I execute the system command of storageLVMThinpoolExists (vgs --noheadings -o lvattr SSD/lxd-ssd-tp) I get twi-a-tz-- which starts with t, so I expect true as result.
Therefore I expect empty to be set to true.

With empty set to true, everything should work.

When I debug the lxc command lxc storage create virt-prod-rub1-01:pool-ssd lvm source=SSD lvm.thinpool_name=lxd-ssd-tp lvm.use_thinpool=true lvm.vg_name=SSD --debug the query to LXD seems ok:

DBUG[02-19|09:37:00] Sending request to LXD                   method=POST url=https://virt-prod-01:8443/1.0/storage-pools etag=
DBUG[02-19|09:37:00] 
	{
		"config": {
			"lvm.thinpool_name": "lxd-ssd-tp",
			"lvm.use_thinpool": "true",
			"lvm.vg_name": "SSD",
			"source": "SSD"
		},
		"description": "",
		"name": "pool-ssd",
		"driver": "lvm"
	} 
Error: Volume group "SSD" is not empty

How can I figure out what went wrong?

10

@tomp I setup the system the 16th January. Then the creation of the storage pools worked. Now I wanted to recreate the pool-ssd as before (I deleted it due to a size misconfiguration)
For the recreation of pool-ssd I moved all containers to pool-hdd which I created identically when I set up the system.

11

Thanks for looking into that.

So that code you linked to is actually the old storage layer that isn’t being used anymore (it will be removed soon). However it does highlight the bug because despite the comment explaining the intention to prevent the use of a non-empty volume group, if the thin pool volume exists it allows it to be used (https://github.com/lxc/lxd/blob/lxd-3.21/lxd/storage_lvm.go#L286-L287).

This was a bug because although the thin pool volumes are separate and multiple ones can coexist in the same volume group, when thin volumes are created in that thin pool they still all share the same namespace, i.e you cannot have 2 thin volumes with the same name exist on different thin pools on the same volume group.

So the bug was fixed in the new storage driver which is here: https://github.com/lxc/lxd/blob/master/lxd/storage/drivers/driver_lvm.go#L200-L204

Assuming you could re-create the storage pool on the existing volume group, was it your intention to use lxd import to re-create the containers in the database?

We have been considering adding an equivalent of the ceph.osd.force_reuse (https://linuxcontainers.org/lxd/docs/master/storage) option but for LVM to allow people who knowingly want to do that, rather than as an accident that may cause unexpected data loss or conflicts down the line.

12

Specifically this part is the fixed bug: https://github.com/lxc/lxd/blob/master/lxd/storage/drivers/driver_lvm.go#L239-L243

13

Thank you for the clarification!

I was unable to recreate a volume group. When I set up the server I created the storage pools pool-ssd and pool-hdd in the thin pools of the non empty VG’s. Due to the misconfiguration of the pool-ssd thinpool size, I moved all containers to the pool-hdd and deleted the pool-ssd. Then the recreation did not work.

A force option for the use of already used volume groups would be great. This would enable the concurrent use of a volume group as long as my manually created logical volumes are not in conflict with the LXD naming scheme.

14

If I understand this correctly, it would allow an administrator to create - via an option - an LVM pool in an existing VG with existing LV, right? That would be excellent, indeed. It would make LXD really suitable for LVM based systems (and not just because of a “bug”), where the basic design concept is to make a physically large storage usable for all apps of a system in a flexible and adaptable way.

Can you estimate when such an option might be available? (Sorry, I don’t want to push, unfortunately I have a huge problem with version 3.20 and obviously can’t revert)

15

I’ve added a pull-request for this feature:

As for timings, the next release should be 2 weeks or so, although @stgraber may be able to cherry pick this into the existing stable branch.

16

Wow, that’s really really fast! Thank you very much!