Can you show me the responses you enter to the questions for joining the member to the cluster?
@tomp here you go:
-bash 5.1 :) sudo /snap/bin/lxd init
WARNING: cgroup v2 is not fully supported yet, proceeding with partial confinement
Would you like to use LXD clustering? (yes/no) [default=no]: yes
What IP address or DNS name should be used to reach this node? [default=10.0.0.11]: 10.0.0.11:8343
Are you joining an existing cluster? (yes/no) [default=no]: yes
Do you have a join token? (yes/no/[token]) [default=no]:
What name should be used to identify this node in the cluster? [default=debixx]:
IP address or FQDN of an existing cluster member: 10.0.0.150:8343
Cluster fingerprint: [redacted]
You can validate this fingerprint by running "lxc info" locally on an existing node.
Is this the correct fingerprint? (yes/no/[fingerprint]) [default=no]: yes
Cluster trust password:
All existing data is lost when joining a cluster, continue? (yes/no) [default=no] yes
Choose "size" property for storage pool "steam": 100GB
Choose "source" property for storage pool "steam":
Choose "parent" property for network "lxdvlan1": eno1
Would you like a YAML "lxd init" preseed to be printed? (yes/no) [default=no]: no
The program then hangs.
@stgraber do you have any ideas on this one?
If not I’ll setup a 3 node cluster with custom listen port and see if I can recreate.
Hmm, no, I don’t have any idea what this may be.
OK so I have setup a similar environment in 3 fresh Ubuntu Focal LXD VMs with the LXD latest/edge snap installed:
v1: 10.128.213.11
v2: 10.128.213.12
v3: 10.128.213.13
Setup LXD on v1:
lxd init
Would you like to use LXD clustering? (yes/no) [default=no]: yes
What IP address or DNS name should be used to reach this node? [default=10.128.213.11]: 10.128.213.11:8343
Are you joining an existing cluster? (yes/no) [default=no]:
What name should be used to identify this node in the cluster? [default=v1]:
Setup password authentication on the cluster? (yes/no) [default=no]:
Do you want to configure a new local storage pool? (yes/no) [default=yes]:
Name of the storage backend to use (btrfs, dir, lvm, zfs) [default=zfs]: dir
Do you want to configure a new remote storage pool? (yes/no) [default=no]:
Would you like to connect to a MAAS server? (yes/no) [default=no]:
Would you like to configure LXD to use an existing bridge or host interface? (yes/no) [default=no]:
Would you like to create a new Fan overlay network? (yes/no) [default=yes]:
What subnet should be used as the Fan underlay? [default=auto]:
Would you like stale cached images to be updated automatically? (yes/no) [default=yes]
Would you like a YAML "lxd init" preseed to be printed? (yes/no) [default=no]:
Add join tokens for v2 and v3:
lxc cluster add v2
eyJzZXJ2ZXJfbmFtZSI6InYyIiwiZmluZ2VycHJpbnQiOiI0YTk1YzE2MTAxOTI5MjBmN2NhNjU4MDNhYjE3MWRmNTdjOTUxMmFhOTFiNDEyOWE1NzFlN2M2NGI3YmI3ZjMxIiwiYWRkcmVzc2VzIjpbIjEwLjEyOC4yMTMuMTE6ODM0MyJdLCJzZWNyZXQiOiJkMjJjMDVhNDlmYWMyYjE4N2VhNjk2NDVlYjMyNTZkMjZiMDhlNjhjYTc3ODU2MjhhZjE4NDA4MTc2ZmZhZjExIn0=
lxc cluster add v3
eyJzZXJ2ZXJfbmFtZSI6InYzIiwiZmluZ2VycHJpbnQiOiI0YTk1YzE2MTAxOTI5MjBmN2NhNjU4MDNhYjE3MWRmNTdjOTUxMmFhOTFiNDEyOWE1NzFlN2M2NGI3YmI3ZjMxIiwiYWRkcmVzc2VzIjpbIjEwLjEyOC4yMTMuMTE6ODM0MyJdLCJzZWNyZXQiOiJhYTY2Mjg5YTQ2NGM5YzJjNWRkNjE4YjA2NDc5ZjEwYWRlZDliZTcxNmNjZTYxNmNhYTQ1ZGZjMjUwYWY5NDQzIn0=
Setup LXD on v2:
lxd init
Would you like to use LXD clustering? (yes/no) [default=no]: yes
What IP address or DNS name should be used to reach this node? [default=10.128.213.12]: 10.128.213.12:8343
Are you joining an existing cluster? (yes/no) [default=no]: yes
Do you have a join token? (yes/no/[token]) [default=no]: eyJzZXJ2ZXJfbmFtZSI6InYyIiwiZmluZ2VycHJpbnQiOiI0YTk1YzE2MTAxOTI5MjBmN2NhNjU4MDNhYjE3MWRmNTdjOTUxMmFhOTFiNDEyOWE1NzFlN2M2NGI3YmI3ZjMxIiwiYWRkcmVzc2VzIjpbIjEwLjEyOC4yMTMuMTE6ODM0MyJdLCJzZWNyZXQiOiJkMjJjMDVhNDlmYWMyYjE4N2VhNjk2NDVlYjMyNTZkMjZiMDhlNjhjYTc3ODU2MjhhZjE4NDA4MTc2ZmZhZjExIn0=
All existing data is lost when joining a cluster, continue? (yes/no) [default=no] yes
Choose "source" property for storage pool "local":
Would you like a YAML "lxd init" preseed to be printed? (yes/no) [default=no]:
Setup LXD on v3:
lxd init
Would you like to use LXD clustering? (yes/no) [default=no]: yes
What IP address or DNS name should be used to reach this node? [default=10.128.213.13]: 10.128.213.13:8343
Are you joining an existing cluster? (yes/no) [default=no]: yes
Do you have a join token? (yes/no/[token]) [default=no]: eyJzZXJ2ZXJfbmFtZSI6InYzIiwiZmluZ2VycHJpbnQiOiI0YTk1YzE2MTAxOTI5MjBmN2NhNjU4MDNhYjE3MWRmNTdjOTUxMmFhOTFiNDEyOWE1NzFlN2M2NGI3YmI3ZjMxIiwiYWRkcmVzc2VzIjpbIjEwLjEyOC4yMTMuMTE6ODM0MyJdLCJzZWNyZXQiOiJhYTY2Mjg5YTQ2NGM5YzJjNWRkNjE4YjA2NDc5ZjEwYWRlZDliZTcxNmNjZTYxNmNhYTQ1ZGZjMjUwYWY5NDQzIn0=
All existing data is lost when joining a cluster, continue? (yes/no) [default=no] yes
Choose "source" property for storage pool "local":
Would you like a YAML "lxd init" preseed to be printed? (yes/no) [default=no]:
On v1:
lxc cluster ls
+------+----------------------------+----------+--------------+----------------+-------------+--------+-------------------+
| NAME | URL | ROLES | ARCHITECTURE | FAILURE DOMAIN | DESCRIPTION | STATE | MESSAGE |
+------+----------------------------+----------+--------------+----------------+-------------+--------+-------------------+
| v1 | https://10.128.213.11:8343 | database | x86_64 | default | | ONLINE | Fully operational |
+------+----------------------------+----------+--------------+----------------+-------------+--------+-------------------+
| v2 | https://10.128.213.12:8343 | database | x86_64 | default | | ONLINE | Fully operational |
+------+----------------------------+----------+--------------+----------------+-------------+--------+-------------------+
| v3 | https://10.128.213.13:8343 | database | x86_64 | default | | ONLINE | Fully operational |
+------+----------------------------+----------+--------------+----------------+-------------+--------+-------------------+
So it seems that at least on a recent version of LXD with a fresh cluster that using a custom port during setup works fine.
What you could try is enabling debug on the new member using:
sudo snap set lxd daemon.debug=true; sudo systemctl reload snap.lxd.daemon
And then doing sudo journalctl -f -g lxd whilst doing a join and see if there are any errors coming up, and paste the output here so we can see what is happening.
Also, at the same time enabling debug on the member you are joining to and capture the same output.