Copy / Move thin pool LV to another array?

LXD
1

I have an LVM volume group on a pair of 4TB drives that are mirrored using mdadm. The volume group hosts several conventional LVs for qemu/kvm VM root filesystems, and my LXD thin pool. I am in the process of moving the LVs to a new mdadm mirror of 1TB SSDs. The physical volume(s) look like this:

# pvs
  PV         VG  Fmt  Attr PSize   PFree   
  /dev/md0   vg0 lvm2 a--  931.38g <578.68g
  /dev/md1   vg0 lvm2 a--   <3.64t   <2.92t

md1 is the old 4TB array, md0 is the new 1TB array

Previously, when I needed to rotate out old disks and move the array (or upgrade the array), I could add the new array to the volume group and mirror the LVs onto the new array, and then drop the old array. This has gone well so far for the regular qemu LVs. I used the command

lvconvert -m 1 vg0/<LVname> /dev/md0

to create the mirrored LV on the new storage. I’ve done this successfully for all the qemu LVs. But I found out this does not work for thin pool LVs. My LVs now look like this:

# lvs -o+devices
  LV                                                                      VG  Attr       LSize   Pool    Origin                                                                  Data%  Meta%  Move Log Cpy%Sync Convert Devices                                            
  LXDPool                                                                 vg0 twi-aotz-- 382.70g                                                                                 5.42   1.30                             LXDPool_tdata(0)                                   
  LXDPool_meta0                                                           vg0 -wi-a-----   1.00g                                                                                                                         /dev/md0(90036)                                    
  LXDPool_meta1                                                           vg0 -wi-a-----   1.00g                                                                                                                         /dev/md1(139951)                                   
  LXDPool_meta2                                                           vg0 -wi-a-----   1.00g                                                                                                                         /dev/md1(140207)                                   
  Windows7                                                                vg0 rwi-a-r---  45.00g                                                                                                        100.00           Windows7_rimage_0(0),Windows7_rimage_1(0)          
  containers_ldap                                                         vg0 Vwi-a-tz--  21.00g LXDPool images_8a825c7097bd5e61c292049503c8d00f68fc98d8b5f5ebe4a45fd844d688aaf9 5.80                                                                                       
  containers_local--nextcloud--server                                     vg0 Vwi-a-tz--  21.00g LXDPool                                                                         8.74                                                                                       
  containers_mailinabox                                                   vg0 Vwi-a-tz--  21.00g LXDPool                                                                         13.46                                                                                      
  containers_mailserver                                                   vg0 Vwi-a-tz--  21.00g LXDPool images_8a825c7097bd5e61c292049503c8d00f68fc98d8b5f5ebe4a45fd844d688aaf9 4.03                                                                                       
  containers_myNCcontainer                                                vg0 Vwi-a-tz--  21.00g LXDPool                                                                         8.72                                                                                       
  containers_mythserver                                                   vg0 Vwi-a-tz--  64.00g LXDPool                                                                         5.41                                                                                       
  containers_nextcloud                                                    vg0 Vwi-aotz--  21.00g LXDPool                                                                         11.46                                                                                      
  containers_roundcube                                                    vg0 Vwi-a-tz--  21.00g LXDPool                                                                         4.91                                                                                       
  containers_test--ui1                                                    vg0 Vwi-a-tz--  21.00g LXDPool images_d18af57022afb992e26314013406e3dca64e458b8a63f0e1668c7eb60038596e 4.09                                                                                       
  couv                                                                    vg0 rwi-a-r---  34.18g                                                                                                        100.00           couv_rimage_0(0),couv_rimage_1(0)                  
  e2fs-cache                                                              vg0 rwi-a-r--- 200.00g                                                                                                        100.00           e2fs-cache_rimage_0(0),e2fs-cache_rimage_1(0)      
  images_84a71299044bc3c3563396bef153c0da83d494f6bf3d38fecc55d776b1e19bf9 vg0 Vwi-a-tz--  21.00g LXDPool                                                                         5.30                                                                                       
  images_8a825c7097bd5e61c292049503c8d00f68fc98d8b5f5ebe4a45fd844d688aaf9 vg0 Vwi-a-tz--  21.00g LXDPool                                                                         3.49                                                                                       
  images_8fa08537ae51c880966626561987153e72d073cbe19dfe5abc062713d929254d vg0 Vwi-a-tz--  10.00g LXDPool                                                                         8.76                                                                                       
  images_a425df004fda0876a060d5b4f76ac790e85c725449273d991939a4cdb179ad83 vg0 Vwi-a-tz--  10.00g LXDPool                                                                         8.76                                                                                       
  images_d18af57022afb992e26314013406e3dca64e458b8a63f0e1668c7eb60038596e vg0 Vwi-a-tz--  21.00g LXDPool                                                                         4.08                                                                                       
  images_ea1d9641ca09f8d7b55548447493ed808113322401861ab1e09d1017e07d4ebd vg0 Vwi-a-tz--  10.00g LXDPool                                                                         9.00                                                                                       
  owncloud                                                                vg0 rwi-aor---  64.00g                                                                                                        100.00           owncloud_rimage_0(0),owncloud_rimage_1(0)          
  squeezeserver                                                           vg0 rwi-aor---   8.50g                                                                                                        100.00           squeezeserver_rimage_0(0),squeezeserver_rimage_1(0)

Hopefully this makes sense (it barely does to me).

What is the recommended procedure to migrate thin pool LVs (perhaps the entire LXDPool itself?) from one physical volume to another? Preferably, I could copy them so I could drop the redundancy later and remove the slower array PV. Note also I have 3 copies of the LXDPool meta data strewn across both md0 and md1 – this was from an earlier (failed) attempt the nearly left my LXD pool complete unusable.

2

I eventually figured this out and will put this note here for whoever stumbles upon this forum post some day. It turns out that lvconvert can manage thin pool volumes. The trick is (was) to use the proper device names. I can’t remember what I tried that didn’t work (really bad at note-taking), but setting all the thinpool volumes to read only (lvchange --activate n <volume>, lvchange --activate y --permission r <volume>) followed by (roughly something like)

sudo lvconvert -m 1 vg0/LXDPool_tmeta /dev/md0
sudo lvconvert -m 1 vg0/LXDPool_tdata /dev/md0
sudo lvconvert -m 0 vg0/LXDPool_tdata /dev/md1
sudo lvconvert -m 0 vg0/LXDPool_tmeta /dev/md1

allowed me to mirror the meta and data volumes of LXDPool onto the second raid array, wait for it to sync, and then remove the first raid array. (I know – the “second” array is md0, the “first” array is md1 – see my earlier post.)

I think what I was doing that didn’t work was trying to move LXDPool directly rather than LXDPool_tmeta and LXDPool_tdata. I had a bunch of related fun moving the many copies of meta I wound up with after failed experiments and actually getting everything off md1. But ultimately, I had everything sync’d to md0 and was able to vgreduce vg0 /dev/md1 and pvremove /dev/md1 - finally the whole mess is on md0. I also had to fix the transaction id count in the metadata because it got screwed up somehow. But LXD started (once I reset the volumes’ permissions to rw) and my containers all seem to work.

The lxblk and lvs -o+devices commands were invaluable in determining where everything was.