How to move a storage to other places?

LXD
#1

Hi!

When I create storage’s they places into /var/lib/lxd/disks/:

lxc storage create ssdpool2 btrfs size=5GB
lxc storage list
+----------+-------------+--------+---------------------------------+---------+
|   NAME   | DESCRIPTION | DRIVER |             SOURCE              | USED BY |
+----------+-------------+--------+---------------------------------+---------+
| ssdpool  |             | btrfs  | /var/lib/lxd/disks/ssdpool.img  | 1       |
+----------+-------------+--------+---------------------------------+---------+
| ssdpool2 |             | btrfs  | /var/lib/lxd/disks/ssdpool2.img | 0       |
+----------+-------------+--------+---------------------------------+---------+

  1. How I can create storage pool in another place, for example create into /mnt/work/hddpool.img ?
    When I use option source I get a message:

    Error: Provided path does not reside on a btrfs filesystem

    Do I need first create loopback file and then formatted into btrfs and then use lxc storage create? Why can’t lxc storage create do it for me?

  2. How can move file /var/lib/lxd/disks/ssdpool2.img to another place, for example to /mnt/work/ssdpool2.img? Create symlink on /var/lib/lxd/disks/ssdpool2.img?

I use latest version of LXD - 3.23

Thanks!

#2

Symlinks won’t work. Using a separate loop device would work, though remember that for production environments, actual partitions will be a lot more reliable and a lot faster.

If you want to move all your disks elsewhere, you can do so while LXD is off, then setup a bind-mount over /var/lib/lxd/disks. Again, stay away from symlinks, those will not work or may break at any time.

#3

Sorry, I do not understand answer on my first question.

I open documentation and see key source for Storage pool configuration:

source - Path to block device or loop file or filesystem entry

Loop file? Ok. I do next steps:

dd if=/dev/zero of=hddpool.img bs=1MiB count=1024
lxc storage create hddpool btrfs source=/mnt/work/lxd/hddpool.img

and I get error:

Error: Provided path does not reside on a btrfs filesystem

Hmm… Ok, I do next:

  1. Attach loopback file to loop device:

    sudo losetup -f -P ./hddpool.img

  2. List of loopdevices:

    losetup -l

NAME       SIZELIMIT OFFSET AUTOCLEAR RO BACK-FILE                      DIO LOG-SEC
/dev/loop1         0      0         0  0 /mnt/work/lxd/hddpool.img        0     512
/dev/loop0         0      0         1  0 /var/lib/lxd/disks/ssdpool.img   0     512

  3. Making a BTRFS:

    sudo mkfs.btrfs /dev/loop1 -L hddpool

btrfs-progs v5.4 
See http://btrfs.wiki.kernel.org for more information.

Label:              hddpool
UUID:               74be420d-f90a-4215-acc1-91b41c971626
Node size:          16384
Sector size:        4096
Filesystem size:    1.00GiB
Block group profiles:
Data:             single            8.00MiB
Metadata:         DUP              51.19MiB
System:           DUP               8.00MiB
SSD detected:       no
Incompat features:  extref, skinny-metadata
Checksum:           crc32c
Number of devices:  1
Devices:
ID        SIZE  PATH
    1     1.00GiB  /dev/loop1

  4. Detach file:

    sudo losetup -D /dev/loop1

  5. Try create storage and get next error:

    lxc storage create hddpool btrfs source=/mnt/work/lxd/hddpool.img

Error: Provided path does not reside on a btrfs filesystem

What I doing wrong?

#4

LXD does not accept loop files outside of /var/lib/lxd/disks, if you use a loop, then you must make sure it will always be using the same path and pass that as the source, not the file backing it.

#5

Ok, I understand. Thank You!

#6

Hello,
May you explain a bit further, please.

I am in the same case scenario, trying to recover containers from an existing btrfs partition.
Following the documentation, it should be feasable to reuse an existing btrfs filesystem like this

mount /dev/sdb3 /media/pvincent/pool1/
lxc storage create pool1 btrfs source=/media/pvincent/pool1/

It replies => Error: Requested btrfs subvolume exists but is not empty

When I type

lxc storage create pool1 btrfs source=/dev/sdb3

It replies => Error: Failed to format block device: Failed to run: mkfs.btrfs /dev/sdb3 -L pool1: /dev/sdb3 appears to contain an existing filesystem (btrfs). Use the -f option to force overwrite.

I understand after mounting this existing storage, the command lxd import will help recovering the existing containers. However, I’m stuck on creating a storage onto an existing btrfs partition.
What did I do wrong ?

btw, I tried something else

  • create a brand new btrfs partition
  • copy from the old container rootfs to the new partition, and then
  • lxd import each container separately => SUCCESS !
    but it’s such a long and painful process. reusing an existing partition would be so helpful !
#7

You can re-use the existing partition, you just need to mount it where it used to go /var/snap/lxd/common/lxd/storage-pools/NAME and then use lxd import on each container.

The first import will cause LXD to re-add the storage pool in the DB for you.

#8

Here are my commands.
May you show me what’s wrong with

FIRST INSTALL + BUILD FRESH /dev/sdb3 as btrfs filesystem

  • snap install lxd
  • . /etc/profile.d/apps-bin-path.sh
  • lxd init
    • Would you like to use an existing block device? => yes
    • Path to the existing block device => /dev/sdb3
      => OK
  • ls -l /var/snap/lxd/common/lxd/storage-pools/
    total 4
drwx--x--x 2 root root 4096 Apr  1 20:58 default
#no mount point, just an empty folder !
  • mount | grep sdb3 # no mount point either !
  • lxc launch images:debian/buster test01
  • lxc list # => test01 created successfully !

STOP LXD, REMOVE ALL except partition /dev/sdb3

  • lxc stop test01
  • snap stop lxd
  • snap remove lxd

REINSTALL LXD, trying to re-use existing partition mounted into default

  • snap install lxd
  • lxd init
    • a new storage pool ? => no
      => OK
  • mkdir /var/snap/lxd/common/lxd/storage-pools/default
  • mount /dev/sdb3 /var/snap/lxd/common/lxd/storage-pools/default
  • ls -l /var/snap/lxd/common/lxd/storage-pools/default
total 0
drwx--x--x 1 root root  12 Apr  1 21:05 containers
drwx--x--x 1 root root   0 Apr  1 20:58 containers-snapshots
drwx--x--x 1 root root   0 Apr  1 20:58 custom
drwx--x--x 1 root root   0 Apr  1 20:58 custom-snapshots
drwx--x--x 1 root root 128 Apr  1 20:58 images
drwx--x--x 1 root root   0 Apr  1 20:58 virtual-machines
drwx--x--x 1 root root   0 Apr  1 20:58 virtual-machines-snapshots
  • lxd import test01
    => Error: The instance “test01” does not seem to exist on any storage pool

CHANGE lxd init, entering default

  • lxd init
    • Create a new BTRFS pool? => no
    • Name of existing BTRFS pool or dataset => default
    • […]
      => Error: Failed to create storage pool ‘default’: Storage pool directory “/var/snap/lxd/common/lxd/storage-pools/default” already exists

UMOUNT /dev/sb3, then lxd init, entering default

  • umount /dev/sdb3
  • rmdir /var/snap/lxd/common/lxd/storage-pools/default/
  • lxd init
    • Create a new BTRFS pool? => no
    • Name of existing BTRFS pool or dataset => default
    • […]
      => Error: Failed to create storage pool ‘default’: Failed to run: btrfs subvolume create /var/snap/lxd/14133/default: ERROR: not a btrfs filesystem: /var/snap/lxd/14133

CHANGE AGAIN lxd init, entering /dev/sdb3

  • lxd init
    • Create a new BTRFS pool? => no
    • Name of the existing BTRFS pool or dataset => /dev/sdb3
    • […]
      => Error: Failed to create storage pool ‘default’: Failed to format block device: Failed to run: mkfs.btrfs /dev/sdb3 -L default: /dev/sdb3 appears to contain an existing filesystem (btrfs).

CHANGE AGAIN lxd init, entering default as a mount point to /dev/sdb3

  • mkdir /var/snap/lxd/common/lxd/storage-pools/default
  • mount /dev/sdb3 /var/snap/lxd/common/lxd/storage-pools/default
  • lxd init
    • Create a new BTRFS pool? (yes/no) => no
    • Name of the existing BTRFS pool or dataset => default
  • Error: Failed to create storage pool ‘default’: Storage pool directory “/var/snap/lxd/common/lxd/storage-pools/default” already exists

No more ideas :frowning:
Need some help please :pray:

#9

That’s because of mount namespaces, not all processes see the same mount entries.

Instead of mount /dev/sdb3 /var/snap/lxd/common/lxd/storage-pools/default, you can use nsenter --mount=/run/snapd/ns/lxd.mnt mount /dev/sdb3 /var/snap/lxd/common/lxd/storage-pools/default which will then mount it in the right namespace

1 Like
#10

:clap: great

I summarize

  • snap install lxd
  • lxd init
    • a new storage pool ? => no
      => OK
  • mkdir /var/snap/lxd/common/lxd/storage-pools/default
  • nsenter --mount=/run/snapd/ns/lxd.mnt mount /dev/sdb3 /var/snap/lxd/common/lxd/storage-pools/default
  • lxd import test01
  • lxc start test01 # => success

Rk: nsenter is the magical command !