Failing to add node to existing cluster with storage_pool of type lvm

LXD
1

I am having problems adding a new node to an existing cluster.
Existing Cluster has

  • two nodes
  • two storage pools:
    local (type: dir) (added during lxd init)
    ssdpool (type: lvm) (added by CLI after cluster of two nodes created)

Output from lxd init run on first node:

config:
  core.https_address: 10.20.16.52:8443
  core.trust_password: ****** (redacted)
networks: []
storage_pools:
- config: {}
  description: ""
  name: local
  driver: dir
profiles:
- config: {}
  description: ""
  devices:
    eth0:
      name: eth0
      nictype: bridged
      parent: br0
      type: nic
    root:
      path: /
      pool: local
      type: disk
  name: default
cluster:
  server_name: devlxd1
  enabled: true
  member_config: []
  cluster_address: ""
  cluster_certificate: ""
  server_address: ""
  cluster_password: ""

Output from lxd init run on second node after joining the cluster hosted by the first node above:

config: {}
networks: []
storage_pools: []
profiles: []
cluster:
  server_name: devlxd2
  enabled: true
  member_config:
  - entity: storage-pool
    name: local
    key: source
    value: ""
    description: '"source" property for storage pool "local"'
  cluster_address: 10.20.16.52:8443
  cluster_certificate: |
    -----BEGIN CERTIFICATE-----
    MIICCDCCAY2gAwIBAgIRAOow6vt74BMJxq5BkdtvQLcwCgYIKoZIzj0EAwMwNTEc
    ... redacted ...
    wYElArIaNEji6+Y9agaNb1HTaifeGbODVXW/FnebIL0sh2+bqVscmJXsVkc=
    -----END CERTIFICATE-----
  server_address: 10.20.16.53:8443
  cluster_password: ***** (redacted)

I added the second storage pool to the nodes, and then the cluster, via CLI:

ptgs-MacBook-Pro:~ pgarneau$ lxc storage create --target devlxd1 ssdpool lvm source=lxdstore
Storage pool ssdpool pending on member devlxd1
ptgs-MacBook-Pro:~ pgarneau$ lxc storage create --target devlxd2 ssdpool lvm source=lxdstore
Storage pool ssdpool pending on member devlxd2
ptgs-MacBook-Pro:~ pgarneau$ lxc storage create ssdpool lvm
Storage pool ssdpool created

After a period of time successfully running the two node cluster, I attempted to add a new node to the cluster. Output from lxd init run on the new node in an attempt to add a third node to the cluster:

ubuntu@devlxd3:~$ sudo lxd init
Would you like to use LXD clustering? (yes/no) [default=no]: yes
What name should be used to identify this node in the cluster? [default=devlxd3]:
What IP address or DNS name should be used to reach this node? [default=10.20.16.70]:
Are you joining an existing cluster? (yes/no) [default=no]: yes
IP address or FQDN of an existing cluster node: 10.20.16.52
Cluster fingerprint: 19e991543d168bc26290243202ad06ca635d3c64c26af1352a0c17ae6ef99ebe
You can validate this fingerprint by running "lxc info" locally on an existing node.
Is this the correct fingerprint? (yes/no) [default=no]: yes
Cluster trust password:
All existing data is lost when joining a cluster, continue? (yes/no) [default=no] yes
Choose "source" property for storage pool "local":
Choose "lvm.vg_name" property for storage pool "ssdpool": lxdstore
Choose "source" property for storage pool "ssdpool": lxdstore
Would you like a YAML "lxd init" preseed to be printed? (yes/no) [default=no]: yes
config: {}
networks: []
storage_pools: []
profiles: []
cluster:
  server_name: devlxd3
  enabled: true
  member_config:
  - entity: storage-pool
    name: local
    key: source
    value: ""
    description: '"source" property for storage pool "local"'
  - entity: storage-pool
    name: ssdpool
    key: lvm.vg_name
    value: lxdstore
    description: '"lvm.vg_name" property for storage pool "ssdpool"'
  - entity: storage-pool
    name: ssdpool
    key: source
    value: lxdstore
    description: '"source" property for storage pool "ssdpool"'
  cluster_address: 10.20.16.52:8443
  cluster_certificate: |
    -----BEGIN CERTIFICATE-----
    MIICCDCCAY2gAwIBAgIRAOow6vt74BMJxq5BkdtvQLcwCgYIKoZIzj0EAwMwNTEc
    ... redacted ...
    wYElArIaNEji6+Y9agaNb1HTaifeGbODVXW/FnebIL0sh2+bqVscmJXsVkc=
    -----END CERTIFICATE-----
  server_address: 10.20.16.70:8443
  cluster_password: ***** (redacted)

The above attempt of lxd init run on the third node failed with the first error message.

First error message (I do not have precise error message warning, but it was fairly close to the following):

Error: Failed to join cluster: Failed to initialize member: Failed to initialize storage pools and networks: Mismatching config for storage pool ssdpool: different values for keys: lvm.thinpool_name

I had not set thinpool_name, and therefore, I did not know what LXD wanted. The online docs state there is a default value for this config item, but it was seemingly not being used by the app. The lxd init process was not asking me for a value for thinpool_name.

I tried these commands to get information, but the command did not return anything:

ptgs-MacBook-Pro:~ pgarneau$ lxc storage get ssdpool lvm.vg_name
ptgs-MacBook-Pro:~ pgarneau$ lxc storage get ssdpool lvm.thinpool_name

I looked for information via Google search, github PRs, github README.md (which led me to lxd sql).
I figured I would need to go to the database since the “lxc storage” commands above were not providing information.

Before working with the database, I tried removing, then reinstalling, lxd on the node I was trying to add to the existing cluster.

sudo apt remove lxd
sudo apt install lxd

I attempted to add the node to the existing cluster. The lxd init process failed again, but I started getting the second error.

Second error message:

Error: Failed to join cluster: Failed to initialize member: Failed to initialize storage pools and networks: Failed to create storage pool 'ssdpool': Volume group "lxdstore" is already used by LXD

I performed a local and global dump, and then looked to the storage_pools_config table.

ubuntu@devlxd1:~$ lxd sql global "select * from storage_pools_config"
+----+-----------------+---------+-------------------------+----------------------------------------------+
| id | storage_pool_id | node_id |           key           |                    value                     |
+----+-----------------+---------+-------------------------+----------------------------------------------+
| 1  | 1               | 1       | source                  | /var/snap/lxd/common/lxd/storage-pools/local |
| 2  | 1               | 2       | source                  | /var/snap/lxd/common/lxd/storage-pools/local |
| 5  | 2               | 1       | source                  | lxdstore                                     |
| 6  | 2               | 1       | volatile.initial_source | lxdstore                                     |
| 7  | 2               | 1       | lvm.vg_name             | lxdstore                                     |
| 8  | 2               | 2       | lvm.vg_name             | lxdstore                                     |
| 9  | 2               | 2       | source                  | lxdstore                                     |
| 10 | 2               | 2       | volatile.initial_source | lxdstore                                     |
+----+-----------------+---------+-------------------------+----------------------------------------------+

I figured the problem was that I did not originally provide setting for lvm.thinpool_name.

Therefore, I added

INSERT INTO storage_pools_config VALUES(11,2,1,'lvm.thinpool_name','LXDThinPool');
INSERT INTO storage_pools_config VALUES(12,2,2,'lvm.thinpool_name','LXDThinPool');

I tried to join the node to the cluster, but still cannot join the node to the cluster.

ubuntu@devlxd3:~$ sudo lxd init
Would you like to use LXD clustering? (yes/no) [default=no]: yes
What name should be used to identify this node in the cluster? [default=devlxd3]:
What IP address or DNS name should be used to reach this node? [default=10.20.16.70]:
Are you joining an existing cluster? (yes/no) [default=no]: yes
IP address or FQDN of an existing cluster node: 10.20.16.52
Cluster fingerprint: 19e991543d168bc26290243202ad06ca635d3c64c26af1352a0c17ae6ef99ebe
You can validate this fingerprint by running "lxc info" locally on an existing node.
Is this the correct fingerprint? (yes/no) [default=no]: yes
Cluster trust password:
All existing data is lost when joining a cluster, continue? (yes/no) [default=no] yes
Choose "lvm.thinpool_name" property for storage pool "ssdpool": LXDThinPool
Choose "lvm.vg_name" property for storage pool "ssdpool": lxdstore
Choose "source" property for storage pool "ssdpool": lxdstore
Choose "source" property for storage pool "local":
Would you like a YAML "lxd init" preseed to be printed? (yes/no) [default=no]: yes
config: {}
networks: []
storage_pools: []
profiles: []
cluster:
  server_name: devlxd3
  enabled: true
  member_config:
  - entity: storage-pool
    name: ssdpool
    key: lvm.thinpool_name
    value: LXDThinPool
    description: '"lvm.thinpool_name" property for storage pool "ssdpool"'
  - entity: storage-pool
    name: ssdpool
    key: lvm.vg_name
    value: lxdstore
    description: '"lvm.vg_name" property for storage pool "ssdpool"'
  - entity: storage-pool
    name: ssdpool
    key: source
    value: lxdstore
    description: '"source" property for storage pool "ssdpool"'
  - entity: storage-pool
    name: local
    key: source
    value: ""
    description: '"source" property for storage pool "local"'
  cluster_address: 10.20.16.52:8443
  cluster_certificate: |
    -----BEGIN CERTIFICATE-----
    MIICCDCCAY2gAwIBAgIRAOow6vt74BMJxq5BkdtvQLcwCgYIKoZIzj0EAwMwNTEc
    ... redacted ...
    wYElArIaNEji6+Y9agaNb1HTaifeGbODVXW/FnebIL0sh2+bqVscmJXsVkc=
    -----END CERTIFICATE-----
  server_address: 10.20.16.70:8443
  cluster_password: ***** (redacted)

Error: Failed to join cluster: Failed to initialize member: Failed to initialize storage pools and networks: Failed to create storage pool 'ssdpool': Volume group "lxdstore" is already used by LXD

Interestingly enough, now the lxd init process is asking me for a value for lvm.thinpool_name.

Now, when I try the same lxc storage commands from above to get information, nothing is returned for lvm.vg_name, but lvm.thinpool_name does return a value:

ptgs-MacBook-Pro:~ pgarneau$lxc storage get ssdpool lvm.vg_name
ptgs-MacBook-Pro:~ pgarneau$lxc storage get ssdpool lvm.thinpool_name
LXDThinPool

I have attempted deleting the two records I added above for lvm.thinpool_name, and now I can recreate the lxd init process not asking for thinpool_name, but the third node still fails to be added to the cluster.

I’ve seen from other posts that the response from lvdisplay was desired. The lvdisplay output is quite lengthy from the first two nodes, but the output for lvdisplay for the node I am trying to add to the cluster is:

ubuntu@devlxd3:~$ sudo lvdisplay
  --- Logical volume ---
  LV Name                LXDThinPool
  VG Name                lxdstore
  LV UUID                oDczAs-A42f-UBJ5-hoYj-bPvq-EL4m-KRlZNk
  LV Write Access        read/write
  LV Creation host, time devlxd3, 2020-03-19 21:55:50 +0000
  LV Pool metadata       LXDThinPool_tmeta
  LV Pool data           LXDThinPool_tdata
  LV Status              available
  # open                 0
  LV Size                693.69 GiB
  Allocated pool data    0.00%
  Allocated metadata     1.84%
  Current LE             177585
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     256
  Block device           253:2

At this point, I am unable to add a 3rd node to the LXD cluster.
Could you help me find the solution to my issues of joining another node to the cluster? I am hoping to join a total of three new nodes to the cluster, but I am stuck right now joining the first node to the original two nodes in the cluster.

I am happy to provide more information if requested.
Thank you.

2

I have read this post: Error when trying to use an existing LVM pool with a new installation

I believe this explains what is currently happening with my LXD cluster.

If this is true, it appears that a cluster cannot have nodes added to it once an LVM storage pool is used. Not being able to add nodes to a cluster due to fear of overwriting an LV is a huge detriment to being able to use a storage_pool of type lvm.

If you’re counting votes, I’ll vote for a flag to overwrite this current barrier of adding nodes to an LXD cluster that uses a storage_pool of type lvm.

3

I took the liberty and added some markdown to your top post so that it is easier to read.
If you do not want that, you may revert the page back to the initial state.