Btrfs pool and regular operations - disk sizes, mount options, monitoring

I’m playing around LXD & dedicated block device for BTRFS, and a bit amused of handling standard ops with such setup, i.e.:

  • not shown in mount lists
  • not shown in lsblk
  • not sure how to check compression ratios
    • i’ve did lxc storage set poolbtrfs btrfs.mount_options 'compress=zstd:3' , but see no options in lxc storage info poolbtrfs
  • as it’s not shown in any mountpoints, not sure how it’s supposed to be monitored (say Nagios, Zabbix …) for disk space/inodes/…

And so on.
I guess it’s on purpose and someone who architect it like that, had ideas behind this on standard operations tools and flow, but i’m not getting that idea (yet).

Would be interesting to know how things supposed to work and other’s experience with BTRFS here.

Hi Roman!

LXD (and other similar software) uses namespaces. You would need to enter the appropriate namespace. Here is how to enter the mount namespace,

sudo nsenter --target $(cat /var/snap/lxd/common/lxd.pid) --mount

Thanks @simos
Yes, I do understand that things live under mount namespaces and not visible from (main, root, primary?) namespace - thus I have a question, how things supposed to integrate with other tooling, for example disk usage monitoring (to ensure I have space on pool).

coming to btrfs question, situation is quite simple - tooling is not available inside that namespace, see:

from host

root@node2:~# ls -lad /usr/sbin/compsize
-rwxr-xr-x 1 root root 22872 Jul 22  2019 /usr/sbin/compsize
root@node2:~# stat /
  File: /
  Size: 4096            Blocks: 8          IO Block: 4096   directory
Device: 900h/2304d      Inode: 2           Links: 26
Access: (0755/drwxr-xr-x)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2021-04-19 07:39:00.839626781 +0200
Modify: 2021-04-16 06:46:13.272769031 +0200
Change: 2021-04-16 06:46:13.272769031 +0200
 Birth: -

inside mount namespace:

root@node2:~# nsenter --target $(cat /var/snap/lxd/common/lxd.pid) --mount
root@node2:/# ls -lad /usr/sbin/compsize
ls: cannot access '/usr/sbin/compsize': No such file or directory
root@node2:/# stat /
  File: /
  Size: 348             Blocks: 0          IO Block: 1024   directory
Device: 701h/1793d      Inode: 10817       Links: 24
Access: (0755/drwxr-xr-x)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2021-01-28 17:49:02.000000000 +0200
Modify: 2021-01-28 17:49:02.000000000 +0200
Change: 2021-01-28 17:49:02.000000000 +0200
 Birth: -

This is a side effect of the snap packaging, rather than something specific to LXD.

For storage pool drivers, such as LVM or ZFS, using the host tooling will show the LXD volumes as normal.

As @simos mentioned you can switch into the LXD snap’s mount namespace, and then because the snap package comes with its own tooling for each storage driver, these are available at a different location, in this case the btrfs tool is at:

/var/lib/snapd/hostfs/snap/lxd/current/bin/btrfs

@stgraber do you have any comment on this topic? Thanks

1 Like

lxc storage info could be used to track the disk usage on the pool.

Possibly the other option you could use is to use an existing BTRFS filesystem mounted on the host and then create a LXD BTRFS storage pool using the existing filesystem using the source property when creating the storage pool. This way the host filesystem should still be able to see the LXD subvolumes.

Here’s an example using a loopback file (not recommended for production setups):

apt install snapd -y
snap install lxd
dd if=/dev/zero of=/home/user/loopbackfile.img bs=100M count=20
sudo losetup -fP /home/user/loopbackfile.img
sudo losetup | grep loopbackfile.img # Get loopback dev name /dev/loopN
apt install btrfs-progs
mkfs.btrfs /dev/loopN
sudo mkdir -p /home/user/btrfs
sudo mount  /dev/loopN /home/user/btrfs
lxc storage create btrfs btrfs source=/home/user/btrfs/lxd
lxc launch images:alpine/3.13 c1 -s btrfs

ls -la /home/user/btrfs/lxd/
total 16
drwxr-xr-x 1 root root 200 Apr 20 08:24 .
drwxr-xr-x 1 root root   6 Apr 20 08:24 ..
drwx--x--x 1 root root   4 Apr 20 08:25 containers
drwx--x--x 1 root root   0 Apr 20 08:24 containers-snapshots
drwx--x--x 1 root root   0 Apr 20 08:24 custom
drwx--x--x 1 root root   0 Apr 20 08:24 custom-snapshots
drwx--x--x 1 root root 128 Apr 20 08:24 images
drwx--x--x 1 root root   0 Apr 20 08:24 virtual-machines
drwx--x--x 1 root root   0 Apr 20 08:24 virtual-machines-snapshots

 ls -la /home/user/btrfs/lxd/containers/c1/rootfs/
total 0
drwxr-xr-x 1 1000000 1000000  114 Apr 19 13:03 .
d--x------ 1 1000000 root      78 Apr 20 08:25 ..
drwxr-xr-x 1 1000000 1000000  878 Apr 19 13:00 bin
drwxr-xr-x 1 1000000 1000000    0 Apr 19 13:03 dev
drwxr-xr-x 1 1000000 1000000  666 Apr 20 08:25 etc
drwxr-xr-x 1 1000000 1000000    0 Jan 14 11:49 home
drwxr-xr-x 1 1000000 1000000  398 Apr 19 13:00 lib
drwxr-xr-x 1 1000000 1000000   28 Jan 14 11:49 media
drwxr-xr-x 1 1000000 1000000    0 Jan 14 11:49 mnt
drwxr-xr-x 1 1000000 1000000    0 Jan 14 11:49 opt
dr-xr-xr-x 1 1000000 1000000    0 Jan 14 11:49 proc
drwx------ 1 1000000 1000000    0 Jan 14 11:49 root
drwxr-xr-x 1 1000000 1000000    0 Jan 14 11:49 run
drwxr-xr-x 1 1000000 1000000 1676 Apr 19 13:00 sbin
drwxr-xr-x 1 1000000 1000000    0 Jan 14 11:49 srv
drwxr-xr-x 1 1000000 1000000    0 Jan 14 11:49 sys
drwxrwxrwt 1 1000000 1000000   36 Apr 20 08:25 tmp
drwxr-xr-x 1 1000000 1000000   54 Apr 19 13:00 usr
drwxr-xr-x 1 1000000 1000000   86 Apr 20 08:25 var

@tomp thanks a lot! I’ll give it a try