LxD cluster create lxd vm with a Jenkins job

We use 4 machines as a LxD clusters providing LxD vm as Jenkins build slaves.
When I use a Jenkins job to delete and re-create a LxD vm, there is an error occurred when doing “lxc config device override oem-iot-focal-1 root size=40GB”:

21:43:40 + lxc config device override oem-iot-focal-1 root size=40GB
21:43:41 Error: Failed to update device “root”: Failed to run: /snap/lxd/19389/bin/sgdisk --move-second-header /var/snap/lxd/common/lxd/storage-pools/local/virtual-machines/oem-iot-focal-1/root.img: Unable to save backup partition table! Perhaps the ‘e’ option on the experts’

Meanwhile it works well when I shell in the machine to do

lxc config device override oem-iot-focal-1 root size=40GB
Device root overridden for oem-iot-focal-1

1 Like

What’s the LXD version? What’s the distribution you’re running this on? What’s the storage pool driver? Is that the LXD snap or some other package?

I use snap lxd v 4.11 on Ubuntu 20.04.2 LTS .
Storage driver: btrfs.

I found the error occurs always on specific node of the cluster.
After I do “lxd cluster remove the_node”, it seems not happen again.
Still need to do more observation.

We’ve seen something similar (although not the same) recently and fixed it:

Are you using a custom image for your instances?

I use images:ubuntu/bionic/cloud, and images:ubuntu/focal/cloud images, and I use cloud-init to do provision to customize our build environment.
This is the steps that we do:

  1. cat oem-iot-focal-x.profile | lxc profile edit oem-iot
  2. lxc init -p oem-iot oem-focal oem-iot-focal-x --vm (oem-focal is the alias of the download image, say images:ubuntu/focal/cloud)
  3. lxc config device override oem-iot-focal-x root size=40GB
  4. lxc start oem-iot-focal-x

OK you shouldn’t be affected by that other issue then, as the image size is <10GB.
Do you have volume.size set on the BTRFS storage pool?

Also have you confirmed you have sufficient space on the BTRFS storage pool to grow the VM to that size on the problem node?

Here is the result of lxc storage info local

info:
description: “”
driver: btrfs
name: local
space used: 13.05GB
total space: 100.00GB
used by:
images:

  • 97ce659460b8dd5667593335a473969a85ba91c4201b6924c892eefb03bf9dc7
    instances:
  • oem-iot-bionic-8g
    profiles:
  • default
  • oem-bto
  • oem-iot
  • oem-pc

So when a LxD vm occupy 40GB, does that mean there is only 60GB left to be assigned to another LxD vm? I remember in virtualbox that space is only the most can be used, and the real space to be used decides run-time. Is that not the case here?

We create a sparse raw block file, but I’m not sure how qemu behaves when it comes to allocating space in that file.

Does it work if you resize to a smaller size?

1 Like

@tomp May I know more detail on how you use “sparse raw block file” to solve the issue?

This also raises a question to my mind: If there are 4 machines add to a single LxD cluster, does the total storage space limit on the smallest disk storage among 4? Is that also the case when we talk about RAM?

For non clustered storage pools (everything except ceph and cephfs) the storage pool is created on each node.

When creating a BTRFS storage pool you have two options; either using an existing partition using the source option or LXD will create a loop-back file of the specified size on your main filesystem.

In the latter case it is expected that each node have the space available to accommodate the storage pool’s loopback file.

Please can you show output of lxc storage show local so I can see how your BTRFS pool is set up.

Did you try creating a VM with a smaller disk as I asked, did it work?

1 Like

@tomp
To your question, yes, when I use smaller disk it works.
Our image build requirements for some projects, however, needs more disk space than default 10G.

Followed by “lxc storage show local”

config: {}
description: ""
name: local
driver: btrfs
used_by:
- /1.0/images/97ce659460b8dd5667593335a473969a85ba91c4201b6924c892eefb03bf9dc7
- /1.0/profiles/default
- /1.0/profiles/oem-iot
status: Created
locations:
- lxd-cluster-3
- core-taipei2
- lxd-cluster-2
- lxd-cluster-1

When I do lxc storage info local on there different codes in the same lxd cluster, I got 3 different results.

node 1:

info:
description: “”
driver: btrfs
name: local
space used: 23.29GB
total space: 100.00GB
used by:
images:

  • 4460c48abe183e5f8107efd0c67e1d5862aca317a55b979afad6b3942fce3e2d
  • 97ce659460b8dd5667593335a473969a85ba91c4201b6924c892eefb03bf9dc7
    instances:
  • oem-iot-focal-1
    profiles:
  • default
  • oem-iot
  • oem-iot-bionic

node 2:

info:
description: “”
driver: btrfs
name: local
space used: 27.94GB
total space: 30.00GB
used by:
images:

  • 97ce659460b8dd5667593335a473969a85ba91c4201b6924c892eefb03bf9dc7
    profiles:
  • default
  • oem-iot
  • oem-iot-bionic

node3:

info:
description: “”
driver: btrfs
name: local
space used: 3.93MB
total space: 30.00GB
used by:
profiles:

  • default
  • oem-iot
  • oem-iot-bionic

@stgraber I have 2 questions:

  1. Where is the btrfs storage located for snap version of LxD? And how can I extend it? Do I need to extend it on every node in the cluster respectively?

  2. When initiate a lxd vm, lxd cluster would find itself one of the node in the cluster to do it. According to my observation, it just takes turns on node1, node2, and node3. Is there any rule for it to balance the load? Or we need to do that manually according to resource on every node? Let’s say we have 3 different RAM/Storage machines in the same LxD cluster.

Please can you run: lxc storage show <pool> --target=<cluster member> for each of the three nodes (replacing <cluster member> for each one please).

This will show you the per-cluster-member pool config, rather than just the non-cluster-member config your previous run of lxc storage show displayed.

This will also help me answer 1.

For question 2. LXD will distribute instances across the cluster at create time or you can specify a particular node at create time using the --target=<cluster member> flag. It uses some fairly basic metrics based on number of instances on each node.

You can see the source code here:

1 Like

For question 1,
Here is the results:

lxc storage show local --target=lxd-cluster-1

config:
size: 100GB
source: /var/snap/lxd/common/lxd/disks/local.img
description: “”
name: local
driver: btrfs
used_by:

  • /1.0/images/4460c48abe183e5f8107efd0c67e1d5862aca317a55b979afad6b3942fce3e2d
  • /1.0/images/97ce659460b8dd5667593335a473969a85ba91c4201b6924c892eefb03bf9dc7
  • /1.0/instances/oem-iot-bionic-8g
  • /1.0/profiles/default
  • /1.0/profiles/oem-iot
  • /1.0/profiles/oem-iot-bionic
    status: Created
    locations:
  • core-taipei2
  • lxd-cluster-2
  • lxd-cluster-1
  • lxd-cluster-3

lxc storage show local --target=lxd-cluster-2

config:
size: 30GB
source: /var/snap/lxd/common/lxd/disks/local.img
description: “”
name: local
driver: btrfs
used_by:

  • /1.0/profiles/default
  • /1.0/profiles/oem-iot
  • /1.0/profiles/oem-iot-bionic
    status: Created
    locations:
  • lxd-cluster-1
  • lxd-cluster-3
  • core-taipei2
  • lxd-cluster-2

lxc storage show local --target=core-taipei2

config:
size: 30GB
source: /var/snap/lxd/common/lxd/disks/local.img
description: “”
name: local
driver: btrfs
used_by:

  • /1.0/images/97ce659460b8dd5667593335a473969a85ba91c4201b6924c892eefb03bf9dc7
  • /1.0/profiles/default
  • /1.0/profiles/oem-iot
  • /1.0/profiles/oem-iot-bionic
    status: Created
    locations:
  • lxd-cluster-2
  • lxd-cluster-1
  • lxd-cluster-3
  • core-taipei2

OK so now we see the problem. The size property on two of the nodes is just 30GB and not 100GB.

This means on 2 of the nodes the BTRFs pool size is only 30GB. This was probably generated automatically for you on storage pool creation based on the free space on the host’s disk if you didn’t specify it.

You can also see that in your setup you have chosen to create a BTRFS storage pool on a LXD generated loopback file (/var/snap/lxd/common/lxd/disks/local.img) so this is emulating a physical disk and is then formatted with BTRFS.

The mounted path for the loopback file is /var/snap/lxd/common/lxd/storage-pools/local however when using the LXD snap package, the mount is hidden inside the snap package’s mount namespace (so looking in that directory directly on the host will show its empty).

Instead you can access it using:

sudo nsenter --mount=/run/snapd/ns/lxd.mnt 

E.g.

sudo nsenter --mount=/run/snapd/ns/lxd.mnt -- mount | grep /var/snap/lxd/common/lxd/storage-pools | grep btrfs
/var/snap/lxd/common/lxd/disks/local.img on /var/snap/lxd/common/lxd/storage-pools/local type btrfs (rw,relatime,space_cache,user_subvol_rm_allowed,subvolid=5,subvol=/)
1 Like

You are not able to resize a storage pool from LXD. If there is nothing you need to keep on the storage pool then I would suggest deleting it and creating a new one of the size you want.

Also, for production usage it is not recommend to use a loopback file, but instead a dedicated existing partition. See https://linuxcontainers.org/lxd/docs/master/storage#loop-disk

E.g. Assuming you already have a BTRFS directory at /some/btrfs/ then you can create a storage pool using it with:

# Specify member specific configs (e.g.`source` could be different for each member):
lxc storage pool create local btrfs source=/some/btrfs --target=<cluster member 1>
lxc storage pool create local btrfs source=/some/btrfs --target=<cluster member 2>
lxc storage pool create local btrfs source=/some/btrfs --target=<cluster member 3>

# Create the storage pool on all members.
lxc storage pool create local btrfs

Alternatively can also use a dedicated empty partition/block device for your BTRFS pool by specifying the block device path with the source property when you create it, and then LXD will create a BTRFS filesystem on the block device for you.

Now I know when join a node into a current cluster, the following need to be indicated instead of using default value.

Choose “size” property for storage pool “local”: 800GB

1 Like
  1. For lxd cluster, is it not good for all nodes in the cluster sharing the same btrfs/zfs pool? So that we can manage the total number without noticing how many spaces left on specific nodes.
  2. Is there a way to use all disk spaces on 3, let’s say 3 nodes in a cluster, as a joint pool? Or we can only share one pool in one specific node in the cluster?

To answer your questions:

  1. ZFS is not a networked filesystem (as far as I know), so it is not possible for LXD running on other machines to access it and treat the combined storage as a single pool.
  2. A LXD instance can only exist on one cluster member at once, so it will use the disk space on the cluster member only, which is why we don’t treat it as consumed on the other cluster members. It really is just a way of referring to the same sort of storage pool by name on each cluster member.

If you want to have a truly shared pool across the cluster members then we support using ceph storage as a networked filesystem.

1 Like