Today I wanted to add a new node to the cluster. Freshly installed Ubuntu 20.04 on that node yesterday. I’m following the same procedure as for the two recent nodes.
$ sudo lxd init
Would you like to use LXD clustering? (yes/no) [default=no]: yes
What name should be used to identify this node in the cluster? [default=roer]:
What IP address or DNS name should be used to reach this node? [default=172.16.16.33]:
Are you joining an existing cluster? (yes/no) [default=no]: yes
IP address or FQDN of an existing cluster node: ijssel.ghs.nl
Cluster fingerprint: f3a7079038205003c4806208104f643ade069877304ac647019c3455320d92a6
You can validate this fingerprint by running "lxc info" locally on an existing node.
Is this the correct fingerprint? (yes/no) [default=no]: yes
Cluster trust password:
All existing data is lost when joining a cluster, continue? (yes/no) [default=no] yes
Choose "lvm.vg_name" property for storage pool "local":
Choose "source" property for storage pool "local": /dev/md1
Would you like a YAML "lxd init" preseed to be printed? (yes/no) [default=no]: yes
config: {}
networks: []
storage_pools: []
profiles: []
cluster:
server_name: roer
enabled: true
member_config:
- entity: storage-pool
name: local
key: lvm.vg_name
value: ""
description: '"lvm.vg_name" property for storage pool "local"'
- entity: storage-pool
name: local
key: source
value: /dev/md1
description: '"source" property for storage pool "local"'
cluster_address: ijssel.ghs.nl:8443
cluster_certificate: |
-----BEGIN CERTIFICATE-----
MIICAzCCAYmgAwIBAgIQNCGwcCf7Asx45h/1jwyJZDAKBggqhkjOPQQDAzA0MRww
...
-----END CERTIFICATE-----
server_address: 172.16.16.33:8443
cluster_password: ...
Error: Failed to join cluster: Failed request to add member: Mismatching config for network lxdbr0: different values for keys: volatile.bridge.hwaddr
OK so this looks like a bug introduced by LXD 4.4 that introduces a stable MAC address across all nodes in a cluster, but too avoid disruption of live nodes we only apply the new stable MAC on network config edit. But this appears to be breaking adding new nodes where the new MAC shouldnt be generated. I’ll look into it.
I’ve got a workaround for you in the meantime. However please do not use it if you’re using the fan networking mode as this might also have a separate issue that I need to look into.
Before adding the new node, ensure existing nodes are upgraded to LXD 4.4 and then on one of the nodes run:
lxc network edit lxdbr0
Then exit out of the edit window without making any changes.
You network will briefly drop and restart your lxdbr0 interface and you should then see a volatile.bridge.hwaddr key in the output of lxc network show lxdbr0.
At this point you should be able to add the new node.
First part was OK, I now have a volatile.bridge.hwaddr on the cluster.
Then I did a reinstall (doing the whole dance of cleaning up the LVM volume group, uninstalling lxd, rebooting, installing lxd). Doing the same lxd init now gave me this error.
Error: Failed to join cluster: Failed to initialize member: Failed to initialize storage pools and networks: Failed to create storage pool 'local': Custom loop file locations are not supported
