Cluster refusing connection from attempted new joining members

goremo · June 30, 2023, 8:33am

I was intending to have an lxd cluster of 5 nodes.
To do this, I have a host machine with virtualbox.
On the virtualbox I created 5 virtual machines:

cd-09-151 192.168.0.151
cd-09-152 192.168.0.152
cd-09-153 192.168.0.153
cd-09-154 192.168.0.154
cd-09-155 192.168.0.155

All the machines are created by the same script so they are identical.
I successfully created initial cluster node via lxd init on 192.168.0.151.
I also successfully joined the cluster from the 2nd node, 192.168.0.152…which can telnet the initial node, 192.168.0.151.
Now the issue is the rest are not able to join or telnet the initial node 192.168.0.151. General error “connection refused”
I would like to understand why this is so and also a guide on how to trouble shoot a case like this.
Below are some specific output including, telnet and ping.

Below is environment specs for initial node and SUCCESSFUL connection to ‘self’ ip address (192.168.0.151) and port 8443.

vagrant@cd-09-151 ~> snap --version
snap    2.59.5
snapd   2.59.5
series  16
ubuntu  22.04
kernel  5.15.0-67-generic
vagrant@cd-09-151 ~> telnet 192.168.0.151 8443
Trying 192.168.0.151...
Connected to 192.168.0.151.
Escape character is '^]'.
^CConnection closed by foreign host.

Below is environment specs for the 2nd node (192.168.0.152) and SUCCESSFUL connection to the initial lxd initial node (192.168.0.151 8443)
Note: this node 192.168.0.152 (cd-09-152) had joined the cluster successfully without any problem.

vagrant@cd-09-152 ~> snap --version
snap    2.59.5
snapd   2.59.5
series  16
ubuntu  22.04
vagrant@cd-09-152 ~> lxd version
5.0.2
vagrant@cd-09-152 ~> telnet 192.168.0.151 8443
Trying 192.168.0.151...
Connected to 192.168.0.151.
Escape character is '^]'.

Below is environment specs for the 3rd node (192.168.0.153) and FAILED connection to the initial lxd initial node (192.168.0.151 8443)
Note that the rest of the nodes 192.168.0.4-5 behave the same.

vagrant@cd-09-153 ~> snap --version
snap    2.59.5
snapd   2.59.5
series  16
ubuntu  22.04
kernel  5.15.0-67-generic
vagrant@cd-09-153 ~> lxd version
5.0.2
vagrant@cd-09-153 ~> telnet 192.168.0.151 8443
Trying 192.168.0.151...
telnet: Unable to connect to remote host: Connection refuse
^CConnection closed by foreign host.

Below is the response for systemctl status snap.lxd.daemon at the initial node:

● snap.lxd.daemon.service - Service for snap application lxd.daemon
     Loaded: loaded (/etc/systemd/system/snap.lxd.daemon.service; static)
     Active: active (running) since Thu 2023-06-29 22:33:20 UTC; 9h ago
TriggeredBy: ● snap.lxd.daemon.unix.socket
   Main PID: 43554 (daemon.start)
      Tasks: 0 (limit: 2233)
     Memory: 19.3M
        CPU: 827ms
     CGroup: /system.slice/snap.lxd.daemon.service
             ‣ 43554 /bin/sh /snap/lxd/24322/commands/daemon.start

Jun 29 22:33:22 cd-09-151 lxd.daemon[43554]: => Starting LXD
Jun 29 22:33:23 cd-09-151 lxd.daemon[43732]: time="2023-06-29T22:33:23Z" level=warning msg=" - Couldn't find the CGroup network priority controller, network priority will be ignored"
Jun 29 22:33:23 cd-09-151 lxd.daemon[43732]: time="2023-06-29T22:33:23Z" level=warning msg="Instance type not operational" driver=qemu err="KVM support is missing (no /dev/kvm)" type=virtual-machine
Jun 29 22:33:26 cd-09-151 lxd.daemon[43554]: => First LXD execution on this system
Jun 29 22:33:26 cd-09-151 lxd.daemon[43554]: => LXD is ready
Jun 29 22:46:08 cd-09-151 lxd.daemon[43732]: time="2023-06-29T22:46:08Z" level=warning msg="Cluster member info not found" address="192.168.0.152:8443"
Jun 29 22:46:09 cd-09-151 lxd.daemon[43732]: time="2023-06-29T22:46:09Z" level=error msg="Unaccounted raft node(s) not found in 'nodes' table for heartbeat: {NodeInfo:{ID:2 Address:192.168.0.152:8443 Role:spare} Name:}"
Jun 29 22:46:10 cd-09-151 lxd.daemon[43732]: time="2023-06-29T22:46:10Z" level=error msg="Error refreshing forkdns" err="Network interface \"lxdfan0\" not found" local="192.168.0.151:8443"
Jun 29 22:51:42 cd-09-151 lxd.daemon[43732]: time="2023-06-29T22:51:42Z" level=warning msg="Cancelling duplicate join token operation" operation=d803717a-a238-419d-a46c-242adf3c3432 serverName=cd-09-153
Jun 29 23:11:36 cd-09-151 lxd.daemon[43732]: time="2023-06-29T23:11:36Z" level=warning msg="Cancelling duplicate join token operation" operation=e77d8ebe-dc50-4ad6-b414-802b723f0d27 serverName=cd-09-153
~
~
~

tomp · June 30, 2023, 9:09am

Seems like you have a local or remote firewall blocking connections.

goremo · June 30, 2023, 9:17am

I had checked on all the nodes including the host using sudo ufw status. All of them are reporting ‘inactive’. Below is test on the initial node and the node that is being ‘refused’.

vagrant@cd-09-151 ~ > sudo ufw status
Status: inactive

vagrant@cd-09-153 ~ > sudo ufw status
Status: inactive

goremo · June 30, 2023, 2:46pm

I recreated the virtual machines and was successfull upto the 3rd node. I then posted issues as resolved. But unfortunately, the issue is still there starting from the 4th cluster member.
Below is output from sudo journalctl -u snap.lxd.daemon:

vagrant@cd-09-151 ~> sudo journalctl -u snap.lxd.daemon
Jun 30 13:56:55 cd-09-151 systemd[1]: Started Service for snap application lxd.daemon.
Jun 30 13:56:55 cd-09-151 lxd.daemon[41895]: => Preparing the system (24322)
Jun 30 13:56:55 cd-09-151 lxd.daemon[41895]: ==> Loading snap configuration
Jun 30 13:56:55 cd-09-151 lxd.daemon[41895]: ==> Creating /var/snap/lxd/common/lxd/logs
Jun 30 13:56:55 cd-09-151 lxd.daemon[41895]: ==> Creating /var/snap/lxd/common/global-conf
Jun 30 13:56:55 cd-09-151 lxd.daemon[41895]: ==> Setting up mntns symlink (mnt:[4026532202])
Jun 30 13:56:55 cd-09-151 lxd.daemon[41895]: ==> Setting up mount propagation on /var/snap/lxd/common/lxd/storage-pools
Jun 30 13:56:55 cd-09-151 lxd.daemon[41895]: ==> Setting up mount propagation on /var/snap/lxd/common/lxd/devices
Jun 30 13:56:55 cd-09-151 lxd.daemon[41895]: ==> Setting up persistent shmounts path
Jun 30 13:56:56 cd-09-151 lxd.daemon[41895]: ====> Making LXD shmounts use the persistent path
Jun 30 13:56:56 cd-09-151 lxd.daemon[41895]: ====> Making LXCFS use the persistent path
Jun 30 13:56:56 cd-09-151 lxd.daemon[41895]: ==> Setting up kmod wrapper
Jun 30 13:56:56 cd-09-151 lxd.daemon[41895]: ==> Preparing /boot
Jun 30 13:56:56 cd-09-151 lxd.daemon[41895]: ==> Preparing a clean copy of /run
Jun 30 13:56:56 cd-09-151 lxd.daemon[41895]: ==> Preparing /run/bin
Jun 30 13:56:56 cd-09-151 lxd.daemon[41895]: ==> Preparing a clean copy of /etc
Jun 30 13:56:56 cd-09-151 lxd.daemon[41895]: ==> Preparing a clean copy of /usr/share/misc
Jun 30 13:56:56 cd-09-151 lxd.daemon[41895]: ==> Setting up ceph configuration
Jun 30 13:56:56 cd-09-151 lxd.daemon[41895]: ==> Setting up LVM configuration
Jun 30 13:56:56 cd-09-151 lxd.daemon[41895]: ==> Setting up OVN configuration
Jun 30 13:56:56 cd-09-151 lxd.daemon[41895]: ==> Rotating logs
Jun 30 13:56:56 cd-09-151 lxd.daemon[41895]: ==> Setting up ZFS (2.1)
Jun 30 13:56:56 cd-09-151 lxd.daemon[41895]: ==> Escaping the systemd cgroups
Jun 30 13:56:56 cd-09-151 lxd.daemon[41895]: ====> Detected cgroup V2
Jun 30 13:56:56 cd-09-151 lxd.daemon[41895]: ==> Escaping the systemd process resource limits
Jun 30 13:56:56 cd-09-151 lxd.daemon[41895]: ==> Increasing the number of inotify user instances
Jun 30 13:56:56 cd-09-151 lxd.daemon[41895]: ==> Increasing the number of keys for a nonroot user
Jun 30 13:56:56 cd-09-151 lxd.daemon[41895]: ==> Increasing the number of bytes for a nonroot user
Jun 30 13:56:56 cd-09-151 lxd.daemon[41895]: ==> Disabling shiftfs on this kernel (auto)
Jun 30 13:56:56 cd-09-151 lxd.daemon[41895]: => Starting LXCFS
Jun 30 13:56:56 cd-09-151 lxd.daemon[42059]: Running constructor lxcfs_init to reload liblxcfs
Jun 30 13:56:56 cd-09-151 lxd.daemon[42059]: mount namespace: 5
Jun 30 13:56:56 cd-09-151 lxd.daemon[42059]: hierarchies:
Jun 30 13:56:56 cd-09-151 lxd.daemon[42059]:   0: fd:   6: cpuset,cpu,io,memory,hugetlb,pids,rdma,misc
Jun 30 13:56:56 cd-09-151 lxd.daemon[42059]: Kernel supports pidfds
Jun 30 13:56:56 cd-09-151 lxd.daemon[42059]: Kernel does not support swap accounting
Jun 30 13:56:56 cd-09-151 lxd.daemon[42059]: api_extensions:
Jun 30 13:56:56 cd-09-151 lxd.daemon[42059]: - cgroups
Jun 30 13:56:56 cd-09-151 lxd.daemon[42059]: - sys_cpu_online
Jun 30 13:56:56 cd-09-151 lxd.daemon[42059]: - proc_cpuinfo
Jun 30 13:56:56 cd-09-151 lxd.daemon[42059]: - proc_diskstats
Jun 30 13:56:56 cd-09-151 lxd.daemon[42059]: - proc_loadavg
Jun 30 13:56:56 cd-09-151 lxd.daemon[42059]: - proc_meminfo
Jun 30 13:56:56 cd-09-151 lxd.daemon[42059]: - proc_stat
Jun 30 13:56:56 cd-09-151 lxd.daemon[42059]: - proc_swaps
Jun 30 13:56:56 cd-09-151 lxd.daemon[42059]: - proc_uptime
Jun 30 13:56:56 cd-09-151 lxd.daemon[42059]: - proc_slabinfo
Jun 30 13:56:56 cd-09-151 lxd.daemon[42059]: - shared_pidns
Jun 30 13:56:56 cd-09-151 lxd.daemon[42059]: - cpuview_daemon
Jun 30 13:56:56 cd-09-151 lxd.daemon[42059]: - loadavg_daemon
Jun 30 13:56:56 cd-09-151 lxd.daemon[42059]: - pidfds
Jun 30 13:56:57 cd-09-151 lxd.daemon[41895]: => Starting LXD
Jun 30 13:56:59 cd-09-151 lxd.daemon[42072]: time="2023-06-30T13:56:59Z" level=warning msg=" - Couldn't find the CGroup network priority controller, network priority will be ignored"
Jun 30 13:56:59 cd-09-151 lxd.daemon[42072]: time="2023-06-30T13:56:59Z" level=warning msg="Instance type not operational" driver=qemu err="KVM support is missing (no /dev/kvm)" type=virtual-machine
Jun 30 13:57:03 cd-09-151 lxd.daemon[41895]: => First LXD execution on this system
Jun 30 13:57:03 cd-09-151 lxd.daemon[41895]: => LXD is ready
Jun 30 14:11:51 cd-09-151 lxd.daemon[42072]: time="2023-06-30T14:11:51Z" level=error msg="Error refreshing forkdns" err="Network interface \"lxdfan0\" not found" local="192.168.0.151:8443"
Jun 30 14:17:09 cd-09-151 lxd.daemon[42072]: time="2023-06-30T14:17:09Z" level=warning msg="Cluster member info not found" address="192.168.0.153:8443"
Jun 30 14:17:09 cd-09-151 lxd.daemon[42072]: time="2023-06-30T14:17:09Z" level=warning msg="Could not rebalance cluster member roles" err="Failed to assign role: a configuration change is already in progress (5>
Jun 30 14:17:09 cd-09-151 lxd.daemon[42072]: time="2023-06-30T14:17:09Z" level=error msg="Error refreshing forkdns" err="Network interface \"lxdfan0\" not found" local="192.168.0.151:8443"
 ESCOC












































ind the CGroup network priority controller, network priority will be ignored"
 not operational" driver=qemu err="KVM support is missing (no /dev/kvm)" type=virtual-machine


g forkdns" err="Network interface \"lxdfan0\" not found" local="192.168.0.151:8443"
r info not found" address="192.168.0.153:8443"
alance cluster member roles" err="Failed to assign role: a configuration change is already in progress (5)" local="192.168.0.151:8443"
g forkdns" err="Network interface \"lxdfan0\" not found" local="192.168.0.151:8443"
 ESCOD
Jun 30 13:56:55 cd-09-151 lxd.daemon[41895]: ==> Setting up persistent shmounts path
Jun 30 13:56:56 cd-09-151 lxd.daemon[41895]: ====> Making LXD shmounts use the persistent path
Jun 30 13:56:56 cd-09-151 lxd.daemon[41895]: ====> Making LXCFS use the persistent path
Jun 30 13:56:56 cd-09-151 lxd.daemon[41895]: ==> Setting up kmod wrapper
Jun 30 13:56:56 cd-09-151 lxd.daemon[41895]: ==> Preparing /boot
Jun 30 13:56:56 cd-09-151 lxd.daemon[41895]: ==> Preparing a clean copy of /run
Jun 30 13:56:56 cd-09-151 lxd.daemon[41895]: ==> Preparing /run/bin
Jun 30 13:56:56 cd-09-151 lxd.daemon[41895]: ==> Preparing a clean copy of /etc
Jun 30 13:56:56 cd-09-151 lxd.daemon[41895]: ==> Preparing a clean copy of /usr/share/misc
Jun 30 13:56:56 cd-09-151 lxd.daemon[41895]: ==> Setting up ceph configuration
Jun 30 13:56:56 cd-09-151 lxd.daemon[41895]: ==> Setting up LVM configuration
Jun 30 13:56:56 cd-09-151 lxd.daemon[41895]: ==> Setting up OVN configuration
Jun 30 13:56:56 cd-09-151 lxd.daemon[41895]: ==> Rotating logs
Jun 30 13:56:56 cd-09-151 lxd.daemon[41895]: ==> Setting up ZFS (2.1)
Jun 30 13:56:56 cd-09-151 lxd.daemon[41895]: ==> Escaping the systemd cgroups
Jun 30 13:56:56 cd-09-151 lxd.daemon[41895]: ====> Detected cgroup V2
Jun 30 13:56:56 cd-09-151 lxd.daemon[41895]: ==> Escaping the systemd process resource limits
Jun 30 13:56:56 cd-09-151 lxd.daemon[41895]: ==> Increasing the number of inotify user instances
Jun 30 13:56:56 cd-09-151 lxd.daemon[41895]: ==> Increasing the number of keys for a nonroot user
Jun 30 13:56:56 cd-09-151 lxd.daemon[41895]: ==> Increasing the number of bytes for a nonroot user
Jun 30 13:56:56 cd-09-151 lxd.daemon[41895]: ==> Disabling shiftfs on this kernel (auto)
Jun 30 13:56:56 cd-09-151 lxd.daemon[41895]: => Starting LXCFS
Jun 30 13:56:56 cd-09-151 lxd.daemon[42059]: Running constructor lxcfs_init to reload liblxcfs
Jun 30 13:56:56 cd-09-151 lxd.daemon[42059]: mount namespace: 5
Jun 30 13:56:56 cd-09-151 lxd.daemon[42059]: hierarchies:
Jun 30 13:56:56 cd-09-151 lxd.daemon[42059]:   0: fd:   6: cpuset,cpu,io,memory,hugetlb,pids,rdma,misc
Jun 30 13:56:56 cd-09-151 lxd.daemon[42059]: Kernel supports pidfds
Jun 30 13:56:56 cd-09-151 lxd.daemon[42059]: Kernel does not support swap accounting
Jun 30 13:56:56 cd-09-151 lxd.daemon[42059]: api_extensions:
Jun 30 13:56:56 cd-09-151 lxd.daemon[42059]: - cgroups
Jun 30 13:56:56 cd-09-151 lxd.daemon[42059]: - sys_cpu_online
Jun 30 13:56:56 cd-09-151 lxd.daemon[42059]: - proc_cpuinfo
Jun 30 13:56:56 cd-09-151 lxd.daemon[42059]: - proc_diskstats
Jun 30 13:56:56 cd-09-151 lxd.daemon[42059]: - proc_loadavg
Jun 30 13:56:56 cd-09-151 lxd.daemon[42059]: - proc_meminfo
Jun 30 13:56:56 cd-09-151 lxd.daemon[42059]: - proc_stat
Jun 30 13:56:56 cd-09-151 lxd.daemon[42059]: - proc_swaps
Jun 30 13:56:56 cd-09-151 lxd.daemon[42059]: - proc_uptime
Jun 30 13:56:56 cd-09-151 lxd.daemon[42059]: - proc_slabinfo
Jun 30 13:56:56 cd-09-151 lxd.daemon[42059]: - shared_pidns
Jun 30 13:56:56 cd-09-151 lxd.daemon[42059]: - cpuview_daemon
Jun 30 13:56:56 cd-09-151 lxd.daemon[42059]: - loadavg_daemon
Jun 30 13:56:56 cd-09-151 lxd.daemon[42059]: - pidfds
Jun 30 13:56:57 cd-09-151 lxd.daemon[41895]: => Starting LXD
Jun 30 13:56:59 cd-09-151 lxd.daemon[42072]: time="2023-06-30T13:56:59Z" level=warning msg=" - Couldn't find the CGroup network priority controller, network priority will be ignored"
Jun 30 13:56:59 cd-09-151 lxd.daemon[42072]: time="2023-06-30T13:56:59Z" level=warning msg="Instance type not operational" driver=qemu err="KVM support is missing (no /dev/kvm)" type=virtual-machine
Jun 30 13:57:03 cd-09-151 lxd.daemon[41895]: => First LXD execution on this system
Jun 30 13:57:03 cd-09-151 lxd.daemon[41895]: => LXD is ready
Jun 30 14:11:51 cd-09-151 lxd.daemon[42072]: time="2023-06-30T14:11:51Z" level=error msg="Error refreshing forkdns" err="Network interface \"lxdfan0\" not found" local="192.168.0.151:8443"
Jun 30 14:17:09 cd-09-151 lxd.daemon[42072]: time="2023-06-30T14:17:09Z" level=warning msg="Cluster member info not found" address="192.168.0.153:8443"
Jun 30 14:17:09 cd-09-151 lxd.daemon[42072]: time="2023-06-30T14:17:09Z" level=warning msg="Could not rebalance cluster member roles" err="Failed to assign role: a configuration change is already in progress (5>
Jun 30 14:17:09 cd-09-151 lxd.daemon[42072]: time="2023-06-30T14:17:09Z" level=error msg="Error refreshing forkdns" err="Network interface \"lxdfan0\" not found" local="192.168.0.151:8443"
 ESCOD
Jun 30 13:56:55 cd-09-151 lxd.daemon[41895]: ==> Setting up persistent shmounts path
Jun 30 13:56:56 cd-09-151 lxd.daemon[41895]: ====> Making LXD shmounts use the persistent path
Jun 30 13:56:56 cd-09-151 lxd.daemon[41895]: ====> Making LXCFS use the persistent path
Jun 30 13:56:56 cd-09-151 lxd.daemon[41895]: ==> Setting up kmod wrapper
Jun 30 13:56:56 cd-09-151 lxd.daemon[41895]: ==> Preparing /boot
Jun 30 13:56:56 cd-09-151 lxd.daemon[41895]: ==> Preparing a clean copy of /run
Jun 30 13:56:56 cd-09-151 lxd.daemon[41895]: ==> Preparing /run/bin
Jun 30 13:56:56 cd-09-151 lxd.daemon[41895]: ==> Preparing a clean copy of /etc
Jun 30 13:56:56 cd-09-151 lxd.daemon[41895]: ==> Preparing a clean copy of /usr/share/misc
Jun 30 13:56:56 cd-09-151 lxd.daemon[41895]: ==> Setting up ceph configuration
Jun 30 13:56:56 cd-09-151 lxd.daemon[41895]: ==> Setting up LVM configuration
Jun 30 13:56:56 cd-09-151 lxd.daemon[41895]: ==> Setting up OVN configuration
Jun 30 13:56:56 cd-09-151 lxd.daemon[41895]: ==> Rotating logs
Jun 30 13:56:56 cd-09-151 lxd.daemon[41895]: ==> Setting up ZFS (2.1)
Jun 30 13:56:56 cd-09-151 lxd.daemon[41895]: ==> Escaping the systemd cgroups
Jun 30 13:56:56 cd-09-151 lxd.daemon[41895]: ====> Detected cgroup V2
Jun 30 13:56:56 cd-09-151 lxd.daemon[41895]: ==> Escaping the systemd process resource limits
Jun 30 13:56:56 cd-09-151 lxd.daemon[41895]: ==> Increasing the number of inotify user instances
Jun 30 13:56:56 cd-09-151 lxd.daemon[41895]: ==> Increasing the number of keys for a nonroot user
Jun 30 13:56:56 cd-09-151 lxd.daemon[41895]: ==> Increasing the number of bytes for a nonroot user
Jun 30 13:56:56 cd-09-151 lxd.daemon[41895]: ==> Disabling shiftfs on this kernel (auto)
Jun 30 13:56:56 cd-09-151 lxd.daemon[41895]: => Starting LXCFS
Jun 30 13:56:56 cd-09-151 lxd.daemon[42059]: Running constructor lxcfs_init to reload liblxcfs
Jun 30 13:56:56 cd-09-151 lxd.daemon[42059]: mount namespace: 5
Jun 30 13:56:56 cd-09-151 lxd.daemon[42059]: hierarchies:
Jun 30 13:56:56 cd-09-151 lxd.daemon[42059]:   0: fd:   6: cpuset,cpu,io,memory,hugetlb,pids,rdma,misc
Jun 30 13:56:56 cd-09-151 lxd.daemon[42059]: Kernel supports pidfds
Jun 30 13:56:56 cd-09-151 lxd.daemon[42059]: Kernel does not support swap accounting
Jun 30 13:56:56 cd-09-151 lxd.daemon[42059]: api_extensions:
Jun 30 13:56:56 cd-09-151 lxd.daemon[42059]: - cgroups
Jun 30 13:56:56 cd-09-151 lxd.daemon[42059]: - sys_cpu_online
Jun 30 13:56:56 cd-09-151 lxd.daemon[42059]: - proc_cpuinfo
Jun 30 13:56:56 cd-09-151 lxd.daemon[42059]: - proc_diskstats
Jun 30 13:56:56 cd-09-151 lxd.daemon[42059]: - proc_loadavg
Jun 30 13:56:56 cd-09-151 lxd.daemon[42059]: - proc_meminfo
Jun 30 13:56:56 cd-09-151 lxd.daemon[42059]: - proc_stat
Jun 30 13:56:56 cd-09-151 lxd.daemon[42059]: - proc_swaps
Jun 30 13:56:56 cd-09-151 lxd.daemon[42059]: - proc_uptime
Jun 30 13:56:56 cd-09-151 lxd.daemon[42059]: - proc_slabinfo
Jun 30 13:56:56 cd-09-151 lxd.daemon[42059]: - shared_pidns
Jun 30 13:56:56 cd-09-151 lxd.daemon[42059]: - cpuview_daemon
Jun 30 13:56:56 cd-09-151 lxd.daemon[42059]: - loadavg_daemon
Jun 30 13:56:56 cd-09-151 lxd.daemon[42059]: - pidfds
Jun 30 13:56:57 cd-09-151 lxd.daemon[41895]: => Starting LXD
Jun 30 13:56:59 cd-09-151 lxd.daemon[42072]: time="2023-06-30T13:56:59Z" level=warning msg=" - Couldn't find the CGroup network priority controller, network priority will be ignored"
Jun 30 13:56:59 cd-09-151 lxd.daemon[42072]: time="2023-06-30T13:56:59Z" level=warning msg="Instance type not operational" driver=qemu err="KVM support is missing (no /dev/kvm)" type=virtual-machine
Jun 30 13:57:03 cd-09-151 lxd.daemon[41895]: => First LXD execution on this system
Jun 30 13:57:03 cd-09-151 lxd.daemon[41895]: => LXD is ready
Jun 30 14:11:51 cd-09-151 lxd.daemon[42072]: time="2023-06-30T14:11:51Z" level=error msg="Error refreshing forkdns" err="Network interface \"lxdfan0\" not found" local="192.168.0.151:8443"
Jun 30 14:17:09 cd-09-151 lxd.daemon[42072]: time="2023-06-30T14:17:09Z" level=warning msg="Cluster member info not found" address="192.168.0.153:8443"
Jun 30 14:17:09 cd-09-151 lxd.daemon[42072]: time="2023-06-30T14:17:09Z" level=warning msg="Could not rebalance cluster member roles" err="Failed to assign role: a configuration change is already in progress (5>
Jun 30 14:17:09 cd-09-151 lxd.daemon[42072]: time="2023-06-30T14:17:09Z" level=error msg="Error refreshing forkdns" err="Network interface \"lxdfan0\" not found" local="192.168.0.151:8443"

tomp · June 30, 2023, 3:27pm

So is LXD being restarted repeatedly? I’m not really following what the logs are saying?

goremo · June 30, 2023, 4:01pm

@tomp, I am also learning the ropes here . Below is output for sudo journalctl -u snap.lxd.daemon -n 20. I hope there could be some clue that can be leading.
FYI: All the 3 members that were successful are intact:

vagrant@cd-09-151 ~> lxc cluster list
+-----------+----------------------------+-----------------+--------------+----------------+-------------+--------+-------------------+
|   NAME    |            URL             |      ROLES      | ARCHITECTURE | FAILURE DOMAIN | DESCRIPTION | STATE  |      MESSAGE      |
+-----------+----------------------------+-----------------+--------------+----------------+-------------+--------+-------------------+
| cd-09-151 | https://192.168.0.151:8443 | database-leader | x86_64       | default        |             | ONLINE | Fully operational |
|           |                            | database        |              |                |             |        |                   |
+-----------+----------------------------+-----------------+--------------+----------------+-------------+--------+-------------------+
| cd-09-152 | https://192.168.0.152:8443 | database        | x86_64       | default        |             | ONLINE | Fully operational |
+-----------+----------------------------+-----------------+--------------+----------------+-------------+--------+-------------------+
| cd-09-153 | https://192.168.0.153:8443 | database        | x86_64       | default        |             | ONLINE | Fully operational |
+-----------+----------------------------+-----------------+--------------+----------------+-------------+--------+-------------------+

But on the logs I am noticing some lines that are sucpecious. May be you can guide.
Example 1(Note 192.168.0.153 is the 3rd member to join):
level=warning msg="Cluster member info not found" address="192.168.0.153:8443"
Example 2:
level=warning msg="Could not rebalance cluster member roles" err="Failed to assign role: a configuration change is already in progress (5)" local="192.168.0.151:8443"
Full output

vagrant@cd-09-151 ~> sudo journalctl -u snap.lxd.daemon -n 20
Jun 30 13:56:56 cd-09-151 lxd.daemon[42059]: - proc_diskstats
Jun 30 13:56:56 cd-09-151 lxd.daemon[42059]: - proc_loadavg
Jun 30 13:56:56 cd-09-151 lxd.daemon[42059]: - proc_meminfo
Jun 30 13:56:56 cd-09-151 lxd.daemon[42059]: - proc_stat
Jun 30 13:56:56 cd-09-151 lxd.daemon[42059]: - proc_swaps
Jun 30 13:56:56 cd-09-151 lxd.daemon[42059]: - proc_uptime
Jun 30 13:56:56 cd-09-151 lxd.daemon[42059]: - proc_slabinfo
Jun 30 13:56:56 cd-09-151 lxd.daemon[42059]: - shared_pidns
Jun 30 13:56:56 cd-09-151 lxd.daemon[42059]: - cpuview_daemon
Jun 30 13:56:56 cd-09-151 lxd.daemon[42059]: - loadavg_daemon
Jun 30 13:56:56 cd-09-151 lxd.daemon[42059]: - pidfds
Jun 30 13:56:57 cd-09-151 lxd.daemon[41895]: => Starting LXD
Jun 30 13:56:59 cd-09-151 lxd.daemon[42072]: time="2023-06-30T13:56:59Z" level=warning msg=" - Couldn't find the CGroup network priority controller, network priority will be ignored"
Jun 30 13:56:59 cd-09-151 lxd.daemon[42072]: time="2023-06-30T13:56:59Z" level=warning msg="Instance type not operational" driver=qemu err="KVM support is missing (no /dev/kvm)" type=virtual-machine
Jun 30 13:57:03 cd-09-151 lxd.daemon[41895]: => First LXD execution on this system
Jun 30 13:57:03 cd-09-151 lxd.daemon[41895]: => LXD is ready
Jun 30 14:11:51 cd-09-151 lxd.daemon[42072]: time="2023-06-30T14:11:51Z" level=error msg="Error refreshing forkdns" err="Network interface \"lxdfan0\" not found" local="192.168.0.151:8443"
Jun 30 14:17:09 cd-09-151 lxd.daemon[42072]: time="2023-06-30T14:17:09Z" level=warning msg="Cluster member info not found" address="192.168.0.153:8443"
Jun 30 14:17:09 cd-09-151 lxd.daemon[42072]: time="2023-06-30T14:17:09Z" level=warning msg="Could not rebalance cluster member roles" err="Failed to assign role: a configuration change is already in progress (5>
Jun 30 14:17:09 cd-09-151 lxd.daemon[42072]: time="2023-06-30T14:17:09Z" level=error msg="Error refreshing forkdns" err="Network interface \"lxdfan0\" not found" local="192.168.0.151:8443"

goremo · July 1, 2023, 2:21pm

I have added port forwarding to vagrant file as per this. I am now able to telnet between the virtual machines inside the host and across multiple hosts(in readiness for lxd cluster).