LXD 4.0/Stable will on load after server restart

I have been running 5 containers on lxc 4.0 for some time.
One of the containers is my file server. I kept loosing connection with my file server so I rebooted the server and now no lxc or lxd commands work.

:~$ systemctl -a
  UNIT                                                                                                           LOAD      ACTIVE   SUB       DESCRIPTION
● boot.automount                                                                                                 not-found inactive dead      boot.automount
  proc-sys-fs-binfmt_misc.automount                                                                              loaded    active   running   Arbitrary Executable File Formats File System Automount Point
  
:~$ snap list
Name    Version        Rev    Tracking       Publisher   Notes
core18  20230530       2785   latest/stable  canonical✓  base
core20  20230613       1950   latest/stable  canonical✓  base
lxd     4.0.9-a29c6f1  24061  4.0/stable     canonical✓  in-cohort
snapd   2.59.5         19457  latest/stable  canonical✓  snapd

~$ snap --version
snap    2.59.5
snapd   2.59.5
series  16
ubuntu  22.04
kernel  5.15.0-76-generic

--~$ snap info lxd|grep -E 'snap-id:|tracking:|installed:'
snap-id:      J60k4JY0HppjwOjW8dZdYc8obXKxujRu
tracking:     4.0/stable
installed:          4.0.9-a29c6f1            (24061)  96MB -
--~$
--~$ modinfo zfs | grep version
version:        2.1.5-1ubuntu6~22.04.1
srcversion:     5A94B4662A7A991696CC35F
vermagic:       5.15.0-76-generic SMP mod_unload modversions

I am dead in the water and need help recovering my containers.

Thanks in advance.

Jeff

Can you show:

  • snap services
  • journalctl -u snap.lxd.daemon -n 300
  • cat /var/snap/lxd/common/lxd/logs/lxd.log

As requested

:~$ snap services
Service       Startup  Current   Notes
lxd.activate  enabled  inactive  -
lxd.daemon    enabled  active    socket-activated

~$ journalctl -u snap.lxd.daemon -n 300
Jun 30 17:27:57 cojoserver lxd.daemon[42353]: t=2023-06-30T17:27:57-0400 lvl=warn msg="Dqlite: attempt 6: server 0: dial: Failed connecting to HTTP endpoint \"0\": dial tcp: address 0: missing port in address"
Jun 30 17:27:58 cojoserver lxd.daemon[42353]: t=2023-06-30T17:27:58-0400 lvl=warn msg="Dqlite: attempt 7: server 0: dial: Failed connecting to HTTP endpoint \"0\": dial tcp: address 0: missing port in address"
Jun 30 17:27:59 cojoserver lxd.daemon[42353]: t=2023-06-30T17:27:59-0400 lvl=warn msg="Dqlite: attempt 8: server 0: dial: Failed connecting to HTTP endpoint \"0\": dial tcp: address 0: missing port in address"
Jun 30 17:28:00 cojoserver lxd.daemon[42353]: t=2023-06-30T17:28:00-0400 lvl=warn msg="Dqlite: attempt 9: server 0: dial: Failed connecting to HTTP endpoint \"0\": dial tcp: address 0: missing port in address"
Jun 30 17:28:01 cojoserver lxd.daemon[42353]: t=2023-06-30T17:28:01-0400 lvl=warn msg="Dqlite: attempt 10: server 0: dial: Failed connecting to HTTP endpoint \"0\": dial tcp: address 0: missing port in address"
Jun 30 17:28:02 cojoserver lxd.daemon[42353]: t=2023-06-30T17:28:02-0400 lvl=warn msg="Dqlite: attempt 11: server 0: dial: Failed connecting to HTTP endpoint \"0\": dial tcp: address 0: missing port in address"
Jun 30 17:28:03 cojoserver lxd.daemon[42353]: t=2023-06-30T17:28:03-0400 lvl=warn msg="Dqlite: attempt 12: server 0: dial: Failed connecting to HTTP endpoint \"0\": dial tcp: address 0: missing port in address"
Jun 30 17:28:06 cojoserver lxd.daemon[42353]: t=2023-06-30T17:28:06-0400 lvl=warn msg="Dqlite: attempt 1: server 0: dial: Failed connecting to HTTP endpoint \"0\": dial tcp: address 0: missing port in address"
Jun 30 17:28:06 cojoserver lxd.daemon[42353]: t=2023-06-30T17:28:06-0400 lvl=warn msg="Dqlite: attempt 2: server 0: dial: Failed connecting to HTTP endpoint \"0\": dial tcp: address 0: missing port in address"
Jun 30 17:28:06 cojoserver lxd.daemon[42353]: t=2023-06-30T17:28:06-0400 lvl=warn msg="Dqlite: attempt 3: server 0: dial: Failed connecting to HTTP endpoint \"0\": dial tcp: address 0: missing port in address"
Jun 30 17:28:07 cojoserver lxd.daemon[42353]: t=2023-06-30T17:28:07-0400 lvl=warn msg="Dqlite: attempt 4: server 0: dial: Failed connecting to HTTP endpoint \"0\": dial tcp: address 0: missing port in address"
Jun 30 17:28:08 cojoserver lxd.daemon[42353]: t=2023-06-30T17:28:08-0400 lvl=warn msg="Dqlite: attempt 5: server 0: dial: Failed connecting to HTTP endpoint \"0\": dial tcp: address 0: missing port in address"
Jun 30 17:28:09 cojoserver lxd.daemon[42353]: t=2023-06-30T17:28:09-0400 lvl=warn msg="Dqlite: attempt 6: server 0: dial: Failed connecting to HTTP endpoint \"0\": dial tcp: address 0: missing port in address"
Jun 30 17:28:10 cojoserver lxd.daemon[42353]: t=2023-06-30T17:28:10-0400 lvl=warn msg="Dqlite: attempt 7: server 0: dial: Failed connecting to HTTP endpoint \"0\": dial tcp: address 0: missing port in address"
Jun 30 17:28:11 cojoserver lxd.daemon[42353]: t=2023-06-30T17:28:11-0400 lvl=warn msg="Dqlite: attempt 8: server 0: dial: Failed connecting to HTTP endpoint \"0\": dial tcp: address 0: missing port in address"
Jun 30 17:28:12 cojoserver lxd.daemon[42353]: t=2023-06-30T17:28:12-0400 lvl=warn msg="Dqlite: attempt 9: server 0: dial: Failed connecting to HTTP endpoint \"0\": dial tcp: address 0: missing port in address"
Jun 30 17:28:13 cojoserver lxd.daemon[42353]: t=2023-06-30T17:28:13-0400 lvl=warn msg="Dqlite: attempt 10: server 0: dial: Failed connecting to HTTP endpoint \"0\": dial tcp: address 0: missing port in address"
Jun 30 17:28:14 cojoserver lxd.daemon[42353]: t=2023-06-30T17:28:14-0400 lvl=warn msg="Dqlite: attempt 11: server 0: dial: Failed connecting to HTTP endpoint \"0\": dial tcp: address 0: missing port in address"
Jun 30 17:28:15 cojoserver lxd.daemon[42353]: t=2023-06-30T17:28:15-0400 lvl=warn msg="Dqlite: attempt 12: server 0: dial: Failed connecting to HTTP endpoint \"0\": dial tcp: address 0: missing port in address"
Jun 30 17:28:18 cojoserver lxd.daemon[42353]: t=2023-06-30T17:28:18-0400 lvl=warn msg="Dqlite: attempt 1: server 0: dial: Failed connecting to HTTP endpoint \"0\": dial tcp: address 0: missing port in address"
Jun 30 17:28:18 cojoserver lxd.daemon[42353]: t=2023-06-30T17:28:18-0400 lvl=warn msg="Dqlite: attempt 2: server 0: dial: Failed connecting to HTTP endpoint \"0\": dial tcp: address 0: missing port in address"
Jun 30 17:28:19 cojoserver lxd.daemon[42353]: t=2023-06-30T17:28:19-0400 lvl=warn msg="Dqlite: attempt 3: server 0: dial: Failed connecting to HTTP endpoint \"0\": dial tcp: address 0: missing port in address"
Jun 30 17:28:19 cojoserver lxd.daemon[42353]: t=2023-06-30T17:28:19-0400 lvl=warn msg="Dqlite: attempt 4: server 0: dial: Failed connecting to HTTP endpoint \"0\": dial tcp: address 0: missing port in address"
Jun 30 17:28:20 cojoserver lxd.daemon[42353]: t=2023-06-30T17:28:20-0400 lvl=warn msg="Dqlite: attempt 5: server 0: dial: Failed connecting to HTTP endpoint \"0\": dial tcp: address 0: missing port in address"
Jun 30 17:28:21 cojoserver lxd.daemon[42353]: t=2023-06-30T17:28:21-0400 lvl=warn msg="Dqlite: attempt 6: server 0: dial: Failed connecting to HTTP endpoint \"0\": dial tcp: address 0: missing port in address"
Jun 30 17:28:22 cojoserver lxd.daemon[42353]: t=2023-06-30T17:28:22-0400 lvl=warn msg="Dqlite: attempt 7: server 0: dial: Failed connecting to HTTP endpoint \"0\": dial tcp: address 0: missing port in address"
Jun 30 17:28:23 cojoserver lxd.daemon[42353]: t=2023-06-30T17:28:23-0400 lvl=warn msg="Dqlite: attempt 8: server 0: dial: Failed connecting to HTTP endpoint \"0\": dial tcp: address 0: missing port in address"
Jun 30 17:28:24 cojoserver lxd.daemon[42353]: t=2023-06-30T17:28:24-0400 lvl=warn msg="Dqlite: attempt 9: server 0: dial: Failed connecting to HTTP endpoint \"0\": dial tcp: address 0: missing port in address"
Jun 30 17:28:25 cojoserver lxd.daemon[42353]: t=2023-06-30T17:28:25-0400 lvl=warn msg="Dqlite: attempt 10: server 0: dial: Failed connecting to HTTP endpoint \"0\": dial tcp: address 0: missing port in address"
Jun 30 17:28:26 cojoserver lxd.daemon[42353]: t=2023-06-30T17:28:26-0400 lvl=warn msg="Dqlite: attempt 11: server 0: dial: Failed connecting to HTTP endpoint \"0\": dial tcp: address 0: missing port in address"
Jun 30 17:28:27 cojoserver lxd.daemon[42353]: t=2023-06-30T17:28:27-0400 lvl=warn msg="Dqlite: attempt 12: server 0: dial: Failed connecting to HTTP endpoint \"0\": dial tcp: address 0: missing port in address"
Jun 30 17:28:28 cojoserver lxd.daemon[42353]: t=2023-06-30T17:28:28-0400 lvl=eror msg="Failed connecting to global database" attempt=2215 err="failed to create dqlite connection: no available dqlite leader server found"
Jun 30 17:28:30 cojoserver lxd.daemon[42353]: t=2023-06-30T17:28:30-0400 lvl=warn msg="Dqlite: attempt 1: server 0: dial: Failed connecting to HTTP endpoint \"0\": dial tcp: address 0: missing port in address"
Jun 30 17:28:31 cojoserver lxd.daemon[42353]: t=2023-06-30T17:28:31-0400 lvl=warn msg="Dqlite: attempt 2: server 0: dial: Failed connecting to HTTP endpoint \"0\": dial tcp: address 0: missing port in address"
Jun 30 17:28:31 cojoserver lxd.daemon[42353]: t=2023-06-30T17:28:31-0400 lvl=warn msg="Dqlite: attempt 3: server 0: dial: Failed connecting to HTTP endpoint \"0\": dial tcp: address 0: missing port in address"
Jun 30 17:28:32 cojoserver lxd.daemon[42353]: t=2023-06-30T17:28:32-0400 lvl=warn msg="Dqlite: attempt 4: server 0: dial: Failed connecting to HTTP endpoint \"0\": dial tcp: address 0: missing port in address"
Jun 30 17:28:33 cojoserver lxd.daemon[42353]: t=2023-06-30T17:28:33-0400 lvl=warn msg="Dqlite: attempt 5: server 0: dial: Failed connecting to HTTP endpoint \"0\": dial tcp: address 0: missing port in address"
lines 1-37...skipping...
Jun 30 17:27:57 cojoserver lxd.daemon[42353]: t=2023-06-30T17:27:57-0400 lvl=warn msg="Dqlite: attempt 6: server 0: dial: Failed connecting to HTTP endpoint \"0\": dial tcp: address 0: missing port in address"
Jun 30 17:27:57 cojoserver lxd.daemon[42353]: t=2023-06-30T17:27:57-0400 lvl=warn msg="Dqlite: attempt 6: server 0: dial: Failed connecting to HTTP endpoint \"0\": dial tcp: address 0: missing port in address"
Jun 30 17:27:58 cojoserver lxd.daemon[42353]: t=2023-06-30T17:27:58-0400 lvl=warn msg="Dqlite: attempt 7: server 0: dial: Failed connecting to HTTP endpoint \"0\": dial tcp: address 0: missing port in address"
Jun 30 17:27:59 cojoserver lxd.daemon[42353]: t=2023-06-30T17:27:59-0400 lvl=warn msg="Dqlite: attempt 8: server 0: dial: Failed connecting to HTTP endpoint \"0\": dial tcp: address 0: missing port in address"
Jun 30 17:28:00 cojoserver lxd.daemon[42353]: t=2023-06-30T17:28:00-0400 lvl=warn msg="Dqlite: attempt 9: server 0: dial: Failed connecting to HTTP endpoint \"0\": dial tcp: address 0: missing port in address"
Jun 30 17:28:01 cojoserver lxd.daemon[42353]: t=2023-06-30T17:28:01-0400 lvl=warn msg="Dqlite: attempt 10: server 0: dial: Failed connecting to HTTP endpoint \"0\": dial tcp: address 0: missing port in address"
Jun 30 17:28:02 cojoserver lxd.daemon[42353]: t=2023-06-30T17:28:02-0400 lvl=warn msg="Dqlite: attempt 11: server 0: dial: Failed connecting to HTTP endpoint \"0\": dial tcp: address 0: missing port in address"
Jun 30 17:28:03 cojoserver lxd.daemon[42353]: t=2023-06-30T17:28:03-0400 lvl=warn msg="Dqlite: attempt 12: server 0: dial: Failed connecting to HTTP endpoint \"0\": dial tcp: address 0: missing port in address"
Jun 30 17:28:06 cojoserver lxd.daemon[42353]: t=2023-06-30T17:28:06-0400 lvl=warn msg="Dqlite: attempt 1: server 0: dial: Failed connecting to HTTP endpoint \"0\": dial tcp: address 0: missing port in address"
Jun 30 17:28:06 cojoserver lxd.daemon[42353]: t=2023-06-30T17:28:06-0400 lvl=warn msg="Dqlite: attempt 2: server 0: dial: Failed connecting to HTTP endpoint \"0\": dial tcp: address 0: missing port in address"
Jun 30 17:28:06 cojoserver lxd.daemon[42353]: t=2023-06-30T17:28:06-0400 lvl=warn msg="Dqlite: attempt 3: server 0: dial: Failed connecting to HTTP endpoint \"0\": dial tcp: address 0: missing port in address"
Jun 30 17:28:07 cojoserver lxd.daemon[42353]: t=2023-06-30T17:28:07-0400 lvl=warn msg="Dqlite: attempt 4: server 0: dial: Failed connecting to HTTP endpoint \"0\": dial tcp: address 0: missing port in address"
Jun 30 17:28:08 cojoserver lxd.daemon[42353]: t=2023-06-30T17:28:08-0400 lvl=warn msg="Dqlite: attempt 5: server 0: dial: Failed connecting to HTTP endpoint \"0\": dial tcp: address 0: missing port in address"
Jun 30 17:28:09 cojoserver lxd.daemon[42353]: t=2023-06-30T17:28:09-0400 lvl=warn msg="Dqlite: attempt 6: server 0: dial: Failed connecting to HTTP endpoint \"0\": dial tcp: address 0: missing port in address"
Jun 30 17:28:10 cojoserver lxd.daemon[42353]: t=2023-06-30T17:28:10-0400 lvl=warn msg="Dqlite: attempt 7: server 0: dial: Failed connecting to HTTP endpoint \"0\": dial tcp: address 0: missing port in address"
Jun 30 17:28:11 cojoserver lxd.daemon[42353]: t=2023-06-30T17:28:11-0400 lvl=warn msg="Dqlite: attempt 8: server 0: dial: Failed connecting to HTTP endpoint \"0\": dial tcp: address 0: missing port in address"
Jun 30 17:28:12 cojoserver lxd.daemon[42353]: t=2023-06-30T17:28:12-0400 lvl=warn msg="Dqlite: attempt 9: server 0: dial: Failed connecting to HTTP endpoint \"0\": dial tcp: address 0: missing port in address"
Jun 30 17:28:13 cojoserver lxd.daemon[42353]: t=2023-06-30T17:28:13-0400 lvl=warn msg="Dqlite: attempt 10: server 0: dial: Failed connecting to HTTP endpoint \"0\": dial tcp: address 0: missing port in address"
Jun 30 17:28:14 cojoserver lxd.daemon[42353]: t=2023-06-30T17:28:14-0400 lvl=warn msg="Dqlite: attempt 11: server 0: dial: Failed connecting to HTTP endpoint \"0\": dial tcp: address 0: missing port in address"
Jun 30 17:28:15 cojoserver lxd.daemon[42353]: t=2023-06-30T17:28:15-0400 lvl=warn msg="Dqlite: attempt 12: server 0: dial: Failed connecting to HTTP endpoint \"0\": dial tcp: address 0: missing port in address"
Jun 30 17:28:18 cojoserver lxd.daemon[42353]: t=2023-06-30T17:28:18-0400 lvl=warn msg="Dqlite: attempt 1: server 0: dial: Failed connecting to HTTP endpoint \"0\": dial tcp: address 0: missing port in address"
Jun 30 17:28:18 cojoserver lxd.daemon[42353]: t=2023-06-30T17:28:18-0400 lvl=warn msg="Dqlite: attempt 2: server 0: dial: Failed connecting to HTTP endpoint \"0\": dial tcp: address 0: missing port in address"
Jun 30 17:28:19 cojoserver lxd.daemon[42353]: t=2023-06-30T17:28:19-0400 lvl=warn msg="Dqlite: attempt 3: server 0: dial: Failed connecting to HTTP endpoint \"0\": dial tcp: address 0: missing port in address"
Jun 30 17:28:19 cojoserver lxd.daemon[42353]: t=2023-06-30T17:28:19-0400 lvl=warn msg="Dqlite: attempt 4: server 0: dial: Failed connecting to HTTP endpoint \"0\": dial tcp: address 0: missing port in address"
Jun 30 17:28:20 cojoserver lxd.daemon[42353]: t=2023-06-30T17:28:20-0400 lvl=warn msg="Dqlite: attempt 5: server 0: dial: Failed connecting to HTTP endpoint \"0\": dial tcp: address 0: missing port in address"
Jun 30 17:28:21 cojoserver lxd.daemon[42353]: t=2023-06-30T17:28:21-0400 lvl=warn msg="Dqlite: attempt 6: server 0: dial: Failed connecting to HTTP endpoint \"0\": dial tcp: address 0: missing port in address"
Jun 30 17:28:22 cojoserver lxd.daemon[42353]: t=2023-06-30T17:28:22-0400 lvl=warn msg="Dqlite: attempt 7: server 0: dial: Failed connecting to HTTP endpoint \"0\": dial tcp: address 0: missing port in address"
Jun 30 17:28:23 cojoserver lxd.daemon[42353]: t=2023-06-30T17:28:23-0400 lvl=warn msg="Dqlite: attempt 8: server 0: dial: Failed connecting to HTTP endpoint \"0\": dial tcp: address 0: missing port in address"
Jun 30 17:28:24 cojoserver lxd.daemon[42353]: t=2023-06-30T17:28:24-0400 lvl=warn msg="Dqlite: attempt 9: server 0: dial: Failed connecting to HTTP endpoint \"0\": dial tcp: address 0: missing port in address"
Jun 30 17:28:25 cojoserver lxd.daemon[42353]: t=2023-06-30T17:28:25-0400 lvl=warn msg="Dqlite: attempt 10: server 0: dial: Failed connecting to HTTP endpoint \"0\": dial tcp: address 0: missing port in address"
Jun 30 17:28:26 cojoserver lxd.daemon[42353]: t=2023-06-30T17:28:26-0400 lvl=warn msg="Dqlite: attempt 11: server 0: dial: Failed connecting to HTTP endpoint \"0\": dial tcp: address 0: missing port in address"
Jun 30 17:28:27 cojoserver lxd.daemon[42353]: t=2023-06-30T17:28:27-0400 lvl=warn msg="Dqlite: attempt 12: server 0: dial: Failed connecting to HTTP endpoint \"0\": dial tcp: address 0: missing port in address"
Jun 30 17:28:28 cojoserver lxd.daemon[42353]: t=2023-06-30T17:28:28-0400 lvl=eror msg="Failed connecting to global database" attempt=2215 err="failed to create dqlite connection: no available dqlite leader server found"
Jun 30 17:28:30 cojoserver lxd.daemon[42353]: t=2023-06-30T17:28:30-0400 lvl=warn msg="Dqlite: attempt 1: server 0: dial: Failed connecting to HTTP endpoint \"0\": dial tcp: address 0: missing port in address"
Jun 30 17:28:31 cojoserver lxd.daemon[42353]: t=2023-06-30T17:28:31-0400 lvl=warn msg="Dqlite: attempt 2: server 0: dial: Failed connecting to HTTP endpoint \"0\": dial tcp: address 0: missing port in address"
Jun 30 17:28:31 cojoserver lxd.daemon[42353]: t=2023-06-30T17:28:31-0400 lvl=warn msg="Dqlite: attempt 3: server 0: dial: Failed connecting to HTTP endpoint \"0\": dial tcp: address 0: missing port in address"
Jun 30 17:28:32 cojoserver lxd.daemon[42353]: t=2023-06-30T17:28:32-0400 lvl=warn msg="Dqlite: attempt 4: server 0: dial: Failed connecting to HTTP endpoint \"0\": dial tcp: address 0: missing port in address"
Jun 30 17:28:33 cojoserver lxd.daemon[42353]: t=2023-06-30T17:28:33-0400 lvl=warn msg="Dqlite: attempt 5: server 0: dial: Failed connecting to HTTP endpoint \"0\": dial tcp: address 0: missing port in address"
Jun 30 17:28:34 cojoserver lxd.daemon[42353]: t=2023-06-30T17:28:34-0400 lvl=warn msg="Dqlite: attempt 6: server 0: dial: Failed connecting to HTTP endpoint \"0\": dial tcp: address 0: missing port in address"
Jun 30 17:28:35 cojoserver lxd.daemon[42353]: t=2023-06-30T17:28:35-0400 lvl=warn msg="Dqlite: attempt 7: server 0: dial: Failed connecting to HTTP endpoint \"0\": dial tcp: address 0: missing port in address"
Jun 30 17:28:36 cojoserver lxd.daemon[42353]: t=2023-06-30T17:28:36-0400 lvl=warn msg="Dqlite: attempt 8: server 0: dial: Failed connecting to HTTP endpoint \"0\": dial tcp: address 0: missing port in address"
Jun 30 17:28:37 cojoserver lxd.daemon[42353]: t=2023-06-30T17:28:37-0400 lvl=warn msg="Dqlite: attempt 9: server 0: dial: Failed connecting to HTTP endpoint \"0\": dial tcp: address 0: missing port in address"
Jun 30 17:28:38 cojoserver lxd.daemon[42353]: t=2023-06-30T17:28:38-0400 lvl=warn msg="Dqlite: attempt 10: server 0: dial: Failed connecting to HTTP endpoint \"0\": dial tcp: address 0: missing port in address"
Jun 30 17:28:39 cojoserver lxd.daemon[42353]: t=2023-06-30T17:28:39-0400 lvl=warn msg="Dqlite: attempt 11: server 0: dial: Failed connecting to HTTP endpoint \"0\": dial tcp: address 0: missing port in address"
Jun 30 17:28:40 cojoserver lxd.daemon[42353]: t=2023-06-30T17:28:40-0400 lvl=warn msg="Dqlite: attempt 12: server 0: dial: Failed connecting to HTTP endpoint \"0\": dial tcp: address 0: missing port in address"
Jun 30 17:28:43 cojoserver lxd.daemon[42353]: t=2023-06-30T17:28:43-0400 lvl=warn msg="Dqlite: attempt 1: server 0: dial: Failed connecting to HTTP endpoint \"0\": dial tcp: address 0: missing port in address"
Jun 30 17:28:43 cojoserver lxd.daemon[42353]: t=2023-06-30T17:28:43-0400 lvl=warn msg="Dqlite: attempt 2: server 0: dial: Failed connecting to HTTP endpoint \"0\": dial tcp: address 0: missing port in address"
Jun 30 17:28:43 cojoserver lxd.daemon[42353]: t=2023-06-30T17:28:43-0400 lvl=warn msg="Dqlite: attempt 3: server 0: dial: Failed connecting to HTTP endpoint \"0\": dial tcp: address 0: missing port in address"
Jun 30 17:28:44 cojoserver lxd.daemon[42353]: t=2023-06-30T17:28:44-0400 lvl=warn msg="Dqlite: attempt 4: server 0: dial: Failed connecting to HTTP endpoint \"0\": dial tcp: address 0: missing port in address"
Jun 30 17:28:45 cojoserver lxd.daemon[42353]: t=2023-06-30T17:28:45-0400 lvl=warn msg="Dqlite: attempt 5: server 0: dial: Failed connecting to HTTP endpoint \"0\": dial tcp: address 0: missing port in address"
Jun 30 17:28:46 cojoserver lxd.daemon[42353]: t=2023-06-30T17:28:46-0400 lvl=warn msg="Dqlite: attempt 6: server 0: dial: Failed connecting to HTTP endpoint \"0\": dial tcp: address 0: missing port in address"
Jun 30 17:28:47 cojoserver lxd.daemon[42353]: t=2023-06-30T17:28:47-0400 lvl=warn msg="Dqlite: attempt 7: server 0: dial: Failed connecting to HTTP endpoint \"0\": dial tcp: address 0: missing port in address"
Jun 30 17:28:48 cojoserver lxd.daemon[42353]: t=2023-06-30T17:28:48-0400 lvl=warn msg="Dqlite: attempt 8: server 0: dial: Failed connecting to HTTP endpoint \"0\": dial tcp: address 0: missing port in address"
Jun 30 17:28:49 cojoserver lxd.daemon[42353]: t=2023-06-30T17:28:49-0400 lvl=warn msg="Dqlite: attempt 9: server 0: dial: Failed connecting to HTTP endpoint \"0\": dial tcp: address 0: missing port in address"
Jun 30 17:28:50 cojoserver lxd.daemon[42353]: t=2023-06-30T17:28:50-0400 lvl=warn msg="Dqlite: attempt 10: server 0: dial: Failed connecting to HTTP endpoint \"0\": dial tcp: address 0: missing port in address"
Jun 30 17:28:51 cojoserver lxd.daemon[42353]: t=2023-06-30T17:28:51-0400 lvl=warn msg="Dqlite: attempt 11: server 0: dial: Failed connecting to HTTP endpoint \"0\": dial tcp: address 0: missing port in address"
Jun 30 17:28:52 cojoserver lxd.daemon[42353]: t=2023-06-30T17:28:52-0400 lvl=warn msg="Dqlite: attempt 12: server 0: dial: Failed connecting to HTTP endpoint \"0\": dial tcp: address 0: missing port in address"
Jun 30 17:28:55 cojoserver lxd.daemon[42353]: t=2023-06-30T17:28:55-0400 lvl=warn msg="Dqlite: attempt 1: server 0: dial: Failed connecting to HTTP endpoint \"0\": dial tcp: address 0: missing port in address"
Jun 30 17:28:55 cojoserver lxd.daemon[42353]: t=2023-06-30T17:28:55-0400 lvl=warn msg="Dqlite: attempt 2: server 0: dial: Failed connecting to HTTP endpoint \"0\": dial tcp: address 0: missing port in address"
Jun 30 17:28:56 cojoserver lxd.daemon[42353]: t=2023-06-30T17:28:56-0400 lvl=warn msg="Dqlite: attempt 3: server 0: dial: Failed connecting to HTTP endpoint \"0\": dial tcp: address 0: missing port in address"
Jun 30 17:28:57 cojoserver lxd.daemon[42353]: t=2023-06-30T17:28:57-0400 lvl=warn msg="Dqlite: attempt 4: server 0: dial: Failed connecting to HTTP endpoint \"0\": dial tcp: address 0: missing port in address"
Jun 30 17:28:58 cojoserver lxd.daemon[42353]: t=2023-06-30T17:28:58-0400 lvl=warn msg="Dqlite: attempt 5: server 0: dial: Failed connecting to HTTP endpoint \"0\": dial tcp: address 0: missing port in address"
Jun 30 17:28:59 cojoserver lxd.daemon[42353]: t=2023-06-30T17:28:59-0400 lvl=warn msg="Dqlite: attempt 6: server 0: dial: Failed connecting to HTTP endpoint \"0\": dial tcp: address 0: missing port in address"
Jun 30 17:29:00 cojoserver lxd.daemon[42353]: t=2023-06-30T17:29:00-0400 lvl=warn msg="Dqlite: attempt 7: server 0: dial: Failed connecting to HTTP endpoint \"0\": dial tcp: address 0: missing port in address"
Jun 30 17:29:01 cojoserver lxd.daemon[42353]: t=2023-06-30T17:29:01-0400 lvl=warn msg="Dqlite: attempt 8: server 0: dial: Failed connecting to HTTP endpoint \"0\": dial tcp: address 0: missing port in address"
Jun 30 17:29:02 cojoserver lxd.daemon[42353]: t=2023-06-30T17:29:02-0400 lvl=warn msg="Dqlite: attempt 9: server 0: dial: Failed connecting to HTTP endpoint \"0\": dial tcp: address 0: missing port in address"

t=2023-06-30T17:05:26-0400 lvl=warn msg="Dqlite: attempt 8: server 0: dial: Failed connecting to HTTP endpoint \"0\": dial tcp: address 0: missing port in address"
t=2023-06-30T17:05:27-0400 lvl=warn msg="Dqlite: attempt 9: server 0: dial: Failed connecting to HTTP endpoint \"0\": dial tcp: address 0: missing port in address"
t=2023-06-30T17:05:28-0400 lvl=warn msg="Dqlite: attempt 10: server 0: dial: Failed connecting to HTTP endpoint \"0\": dial tcp: address 0: missing port in address"
t=2023-06-30T17:05:29-0400 lvl=warn msg="Dqlite: attempt 11: server 0: dial: Failed connecting to HTTP endpoint \"0\": dial tcp: address 0: missing port in address"
t=2023-06-30T17:05:30-0400 lvl=warn msg="Dqlite: attempt 12: server 0: dial: Failed connecting to HTTP endpoint \"0\": dial tcp: address 0: missing port in address"
t=2023-06-30T17:05:33-0400 lvl=info msg="Connecting to global database"
t=2023-06-30T17:05:33-0400 lvl=warn msg="Dqlite: attempt 1: server 0: dial: Failed connecting to HTTP endpoint \"0\": dial tcp: address 0: missing port in address"
t=2023-06-30T17:05:33-0400 lvl=warn msg="Dqlite: attempt 2: server 0: dial: Failed connecting to HTTP endpoint \"0\": dial tcp: address 0: missing port in address"
t=2023-06-30T17:05:34-0400 lvl=warn msg="Dqlite: attempt 3: server 0: dial: Failed connecting to HTTP endpoint \"0\": dial tcp: address 0: missing port in address"
t=2023-06-30T17:05:34-0400 lvl=warn msg="Dqlite: attempt 4: server 0: dial: Failed connecting to HTTP endpoint \"0\": dial tcp: address 0: missing port in address"
t=2023-06-30T17:05:35-0400 lvl=warn msg="Dqlite: attempt 5: server 0: dial: Failed connecting to HTTP endpoint \"0\": dial tcp: address 0: missing port in address"
t=2023-06-30T17:05:36-0400 lvl=warn msg="Dqlite: attempt 6: server 0: dial: Failed connecting to HTTP endpoint \"0\": dial tcp: address 0: missing port in address"
t=2023-06-30T17:05:37-0400 lvl=warn msg="Dqlite: attempt 7: server 0: dial: Failed connecting to HTTP endpoint \"0\": dial tcp: address 0: missing port in address"
t=2023-06-30T17:05:38-0400 lvl=warn msg="Dqlite: attempt 8: server 0: dial: Failed connecting to HTTP endpoint \"0\": dial tcp: address 0: missing port in address"
t=2023-06-30T17:05:39-0400 lvl=warn msg="Dqlite: attempt 9: server 0: dial: Failed connecting to HTTP endpoint \"0\": dial tcp: address 0: missing port in address"
t=2023-06-30T17:05:40-0400 lvl=warn msg="Dqlite: attempt 10: server 0: dial: Failed connecting to HTTP endpoint \"0\": dial tcp: address 0: missing port in address"
t=2023-06-30T17:05:41-0400 lvl=warn msg="Dqlite: attempt 11: server 0: dial: Failed connecting to HTTP endpoint \"0\": dial tcp: address 0: missing port in address"
t=2023-06-30T17:05:42-0400 lvl=warn msg="Dqlite: attempt 12: server 0: dial: Failed connecting to HTTP endpoint \"0\": dial tcp: address 0: missing port in address"
t=2023-06-30T17:05:43-0400 lvl=eror msg="Failed connecting to global database" attempt=2105 err="failed to create dqlite connection: no available dqlite leader server found"
t=2023-06-30T17:05:45-0400 lvl=info msg="Connecting to global database"
t=2023-06-30T17:05:45-0400 lvl=warn msg="Dqlite: attempt 1: server 0: dial: Failed connecting to HTTP endpoint \"0\": dial tcp: address 0: missing port in address"
t=2023-06-30T17:05:46-0400 lvl=warn msg="Dqlite: attempt 2: server 0: dial: Failed connecting to HTTP endpoint \"0\": dial tcp: address 0: missing port in address"
t=2023-06-30T17:05:46-0400 lvl=warn msg="Dqlite: attempt 3: server 0: dial: Failed connecting to HTTP endpoint \"0\": dial tcp: address 0: missing port in address"
t=2023-06-30T17:05:47-0400 lvl=warn msg="Dqlite: attempt 4: server 0: dial: Failed connecting to HTTP endpoint \"0\": dial tcp: address 0: missing port in address"
t=2023-06-30T17:05:48-0400 lvl=warn msg="Dqlite: attempt 5: server 0: dial: Failed connecting to HTTP endpoint \"0\": dial tcp: address 0: missing port in address"
t=2023-06-30T17:05:49-0400 lvl=warn msg="Dqlite: attempt 6: server 0: dial: Failed connecting to HTTP endpoint \"0\": dial tcp: address 0: missing port in address"
t=2023-06-30T17:05:50-0400 lvl=warn msg="Dqlite: attempt 7: server 0: dial: Failed connecting to HTTP endpoint \"0\": dial tcp: address 0: missing port in address"
t=2023-06-30T17:05:51-0400 lvl=warn msg="Dqlite: attempt 8: server 0: dial: Failed connecting to HTTP endpoint \"0\": dial tcp: address 0: missing port in address"
t=2023-06-30T17:05:52-0400 lvl=warn msg="Dqlite: attempt 9: server 0: dial: Failed connecting to HTTP endpoint \"0\": dial tcp: address 0: missing port in address"
t=2023-06-30T17:05:53-0400 lvl=warn msg="Dqlite: attempt 10: server 0: dial: Failed connecting to HTTP endpoint \"0\": dial tcp: address 0: missing port in address"
t=2023-06-30T17:05:54-0400 lvl=warn msg="Dqlite: attempt 11: server 0: dial: Failed connecting to HTTP endpoint \"0\": dial tcp: address 0: missing port in address"
t=2023-06-30T17:05:55-0400 lvl=warn msg="Dqlite: attempt 12: server 0: dial: Failed connecting to HTTP endpoint \"0\": dial tcp: address 0: missing port in address"
t=2023-06-30T17:05:58-0400 lvl=info msg="Connecting to global database"
t=2023-06-30T17:05:58-0400 lvl=warn msg="Dqlite: attempt 1: server 0: dial: Failed connecting to HTTP endpoint \"0\": dial tcp: address 0: missing port in address"
t=2023-06-30T17:05:58-0400 lvl=warn msg="Dqlite: attempt 2: server 0: dial: Failed connecting to HTTP endpoint \"0\": dial tcp: address 0: missing port in address"
t=2023-06-30T17:05:58-0400 lvl=warn msg="Dqlite: attempt 3: server 0: dial: Failed connecting to HTTP endpoint \"0\": dial tcp: address 0: missing port in address"
t=2023-06-30T17:05:59-0400 lvl=warn msg="Dqlite: attempt 4: server 0: dial: Failed connecting to HTTP endpoint \"0\": dial tcp: address 0: missing port in address"
t=2023-06-30T17:06:00-0400 lvl=warn msg="Dqlite: attempt 5: server 0: dial: Failed connecting to HTTP endpoint \"0\": dial tcp: address 0: missing port in address"
t=2023-06-30T17:06:01-0400 lvl=warn msg="Dqlite: attempt 6: server 0: dial: Failed connecting to HTTP endpoint \"0\": dial tcp: address 0: missing port in address"
t=2023-06-30T17:06:02-0400 lvl=warn msg="Dqlite: attempt 7: server 0: dial: Failed connecting to HTTP endpoint \"0\": dial tcp: address 0: missing port in address"
t=2023-06-30T17:06:03-0400 lvl=warn msg="Dqlite: attempt 8: server 0: dial: Failed connecting to HTTP endpoint \"0\": dial tcp: address 0: missing port in address"
t=2023-06-30T17:06:04-0400 lvl=warn msg="Dqlite: attempt 9: server 0: dial: Failed connecting to HTTP endpoint \"0\": dial tcp: address 0: missing port in address"
t=2023-06-30T17:06:05-0400 lvl=warn msg="Dqlite: attempt 10: server 0: dial: Failed connecting to HTTP endpoint \"0\": dial tcp: address 0: missing port in address"
t=2023-06-30T17:06:06-0400 lvl=warn msg="Dqlite: attempt 11: server 0: dial: Failed connecting to HTTP endpoint \"0\": dial tcp: address 0: missing port in address"
t=2023-06-30T17:06:07-0400 lvl=warn msg="Dqlite: attempt 12: server 0: dial: Failed connecting to HTTP endpoint \"0\": dial tcp: address 0: missing port in address"
t=2023-06-30T17:06:10-0400 lvl=info msg="Connecting to global database"
t=2023-06-30T17:06:10-0400 lvl=warn msg="Dqlite: attempt 1: server 0: dial: Failed connecting to HTTP endpoint \"0\": dial tcp: address 0: missing port in address"
t=2023-06-30T17:06:10-0400 lvl=warn msg="Dqlite: attempt 2: server 0: dial: Failed connecting to HTTP endpoint \"0\": dial tcp: address 0: missing port in address"
t=2023-06-30T17:06:11-0400 lvl=warn msg="Dqlite: attempt 3: server 0: dial: Failed connecting to HTTP endpoint \"0\": dial tcp: address 0: missing port in address"
t=2023-06-30T17:06:12-0400 lvl=warn msg="Dqlite: attempt 4: server 0: dial: Failed connecting to HTTP endpoint \"0\": dial tcp: address 0: missing port in address"
t=2023-06-30T17:06:13-0400 lvl=warn msg="Dqlite: attempt 5: server 0: dial: Failed connecting to HTTP endpoint \"0\": dial tcp: address 0: missing port in address"
t=2023-06-30T17:06:14-0400 lvl=warn msg="Dqlite: attempt 6: server 0: dial: Failed connecting to HTTP endpoint \"0\": dial tcp: address 0: missing port in address"
t=2023-06-30T17:06:15-0400 lvl=warn msg="Dqlite: attempt 7: server 0: dial: Failed connecting to HTTP endpoint \"0\": dial tcp: address 0: missing port in address"
t=2023-06-30T17:06:16-0400 lvl=warn msg="Dqlite: attempt 8: server 0: dial: Failed connecting to HTTP endpoint \"0\": dial tcp: address 0: missing port in address"
t=2023-06-30T17:06:17-0400 lvl=warn msg="Dqlite: attempt 9: server 0: dial: Failed connecting to HTTP endpoint \"0\": dial tcp: address 0: missing port in address"
t=2023-06-30T17:06:18-0400 lvl=warn msg="Dqlite: attempt 10: server 0: dial: Failed connecting to HTTP endpoint \"0\": dial tcp: address 0: missing port in address"
t=2023-06-30T17:06:19-0400 lvl=warn msg="Dqlite: attempt 11: server 0: dial: Failed connecting to HTTP endpoint \"0\": dial tcp: address 0: missing port in address"
t=2023-06-30T17:06:20-0400 lvl=warn msg="Dqlite: attempt 12: server 0: dial: Failed connecting to HTTP endpoint \"0\": dial tcp: address 0: missing port in address"
t=2023-06-30T17:06:23-0400 lvl=info msg="Connecting to global database"
t=2023-06-30T17:06:23-0400 lvl=warn msg="Dqlite: attempt 1: server 0: dial: Failed connecting to HTTP endpoint \"0\": dial tcp: address 0: missing port in address"
t=2023-06-30T17:06:23-0400 lvl=warn msg="Dqlite: attempt 2: server 0: dial: Failed connecting to HTTP endpoint \"0\": dial tcp: address 0: missing port in address"
t=2023-06-30T17:06:23-0400 lvl=warn msg="Dqlite: attempt 3: server 0: dial: Failed connecting to HTTP endpoint \"0\": dial tcp: address 0: missing port in address"
t=2023-06-30T17:06:24-0400 lvl=warn msg="Dqlite: attempt 4: server 0: dial: Failed connecting to HTTP endpoint \"0\": dial tcp: address 0: missing port in address"
t=2023-06-30T17:06:25-0400 lvl=warn msg="Dqlite: attempt 5: server 0: dial: Failed connecting to HTTP endpoint \"0\": dial tcp: address 0: missing port in address"
t=2023-06-30T17:06:26-0400 lvl=warn msg="Dqlite: attempt 6: server 0: dial: Failed connecting to HTTP endpoint \"0\": dial tcp: address 0: missing port in address"
t=2023-06-30T17:06:27-0400 lvl=warn msg="Dqlite: attempt 7: server 0: dial: Failed connecting to HTTP endpoint \"0\": dial tcp: address 0: missing port in address"
t=2023-06-30T17:06:28-0400 lvl=warn msg="Dqlite: attempt 8: server 0: dial: Failed connecting to HTTP endpoint \"0\": dial tcp: address 0: missing port in address"
t=2023-06-30T17:06:29-0400 lvl=warn msg="Dqlite: attempt 9: server 0: dial: Failed connecting to HTTP endpoint \"0\": dial tcp: address 0: missing port in address"
t=2023-06-30T17:06:30-0400 lvl=warn msg="Dqlite: attempt 10: server 0: dial: Failed connecting to HTTP endpoint \"0\": dial tcp: address 0: missing port in address"
t=2023-06-30T17:06:31-0400 lvl=warn msg="Dqlite: attempt 11: server 0: dial: Failed connecting to HTTP endpoint \"0\": dial tcp: address 0: missing port in address"
t=2023-06-30T17:06:32-0400 lvl=warn msg="Dqlite: attempt 12: server 0: dial: Failed connecting to HTTP endpoint \"0\": dial tcp: address 0: missing port in address"
t=2023-06-30T17:06:35-0400 lvl=info msg="Connecting to global database"
t=2023-06-30T17:06:35-0400 lvl=warn msg="Dqlite: attempt 1: server 0: dial: Failed connecting to HTTP endpoint \"0\": dial tcp: address 0: missing port in address"
t=2023-06-30T17:06:35-0400 lvl=warn msg="Dqlite: attempt 2: server 0: dial: Failed connecting to HTTP endpoint \"0\": dial tcp: address 0: missing port in address"
t=2023-06-30T17:06:36-0400 lvl=warn msg="Dqlite: attempt 3: server 0: dial: Failed connecting to HTTP endpoint \"0\": dial tcp: address 0: missing port in address"
t=2023-06-30T17:06:36-0400 lvl=warn msg="Dqlite: attempt 4: server 0: dial: Failed connecting to HTTP endpoint \"0\": dial tcp: address 0: missing port in address"
t=2023-06-30T17:06:37-0400 lvl=warn msg="Dqlite: attempt 5: server 0: dial: Failed connecting to HTTP endpoint \"0\": dial tcp: address 0: missing port in address"
t=2023-06-30T17:06:38-0400 lvl=warn msg="Dqlite: attempt 6: server 0: dial: Failed connecting to HTTP endpoint \"0\": dial tcp: address 0: missing port in address"
t=2023-06-30T17:06:39-0400 lvl=warn msg="Dqlite: attempt 7: server 0: dial: Failed connecting to HTTP endpoint \"0\": dial tcp: address 0: missing port in address"
t=2023-06-30T17:06:40-0400 lvl=warn msg="Dqlite: attempt 8: server 0: dial: Failed connecting to HTTP endpoint \"0\": dial tcp: address 0: missing port in address"
t=2023-06-30T17:06:41-0400 lvl=warn msg="Dqlite: attempt 9: server 0: dial: Failed connecting to HTTP endpoint \"0\": dial tcp: address 0: missing port in address"
t=2023-06-30T17:06:42-0400 lvl=warn msg="Dqlite: attempt 10: server 0: dial: Failed connecting to HTTP endpoint \"0\": dial tcp: address 0: missing port in address"
t=2023-06-30T17:06:43-0400 lvl=warn msg="Dqlite: attempt 11: server 0: dial: Failed connecting to HTTP endpoint \"0\": dial tcp: address 0: missing port in address"
t=2023-06-30T17:06:44-0400 lvl=warn msg="Dqlite: attempt 12: server 0: dial: Failed connecting to HTTP endpoint \"0\": dial tcp: address 0: missing port in address"
t=2023-06-30T17:06:45-0400 lvl=eror msg="Failed connecting to global database" attempt=2110 err="failed to create dqlite connection: no available dqlite leader server found"
t=2023-06-30T17:06:47-0400 lvl=info msg="Connecting to global database"
t=2023-06-30T17:06:47-0400 lvl=warn msg="Dqlite: attempt 1: server 0: dial: Failed connecting to HTTP endpoint \"0\": dial tcp: address 0: missing port in address"
t=2023-06-30T17:06:48-0400 lvl=warn msg="Dqlite: attempt 2: server 0: dial: Failed connecting to HTTP endpoint \"0\": dial tcp: address 0: missing port in address"
t=2023-06-30T17:06:48-0400 lvl=warn msg="Dqlite: attempt 3: server 0: dial: Failed connecting to HTTP endpoint \"0\": dial tcp: address 0: missing port in address"
t=2023-06-30T17:06:49-0400 lvl=warn msg="Dqlite: attempt 4: server 0: dial: Failed connecting to HTTP endpoint \"0\": dial tcp: address 0: missing port in address"
t=2023-06-30T17:06:50-0400 lvl=warn msg="Dqlite: attempt 5: server 0: dial: Failed connecting to HTTP endpoint \"0\": dial tcp: address 0: missing port in address"
t=2023-06-30T17:06:51-0400 lvl=warn msg="Dqlite: attempt 6: server 0: dial: Failed connecting to HTTP endpoint \"0\": dial tcp: address 0: missing port in address"
t=2023-06-30T17:06:52-0400 lvl=warn msg="Dqlite: attempt 7: server 0: dial: Failed connecting to HTTP endpoint \"0\": dial tcp: address 0: missing port in address"
t=2023-06-30T17:06:53-0400 lvl=warn msg="Dqlite: attempt 8: server 0: dial: Failed connecting to HTTP endpoint \"0\": dial tcp: address 0: missing port in address"
t=2023-06-30T17:06:54-0400 lvl=warn msg="Dqlite: attempt 9: server 0: dial: Failed connecting to HTTP endpoint \"0\": dial tcp: address 0: missing port in address"
t=2023-06-30T17:06:55-0400 lvl=warn msg="Dqlite: attempt 10: server 0: dial: Failed connecting to HTTP endpoint \"0\": dial tcp: address 0: missing port in address"
t=2023-06-30T17:06:56-0400 lvl=warn msg="Dqlite: attempt 11: server 0: dial: Failed connecting to HTTP endpoint \"0\": dial tcp: address 0: missing port in address"
t=2023-06-30T17:06:57-0400 lvl=warn msg="Dqlite: attempt 12: server 0: dial: Failed connecting to HTTP endpoint \"0\": dial tcp: address 0: missing port in address"
t=2023-06-30T17:07:00-0400 lvl=info msg="Connecting to global database"
t=2023-06-30T17:07:00-0400 lvl=warn msg="Dqlite: attempt 1: server 0: dial: Failed connecting to HTTP endpoint \"0\": dial tcp: address 0: missing port in address"
t=2023-06-30T17:07:00-0400 lvl=warn msg="Dqlite: attempt 2: server 0: dial: Failed connecting to HTTP endpoint \"0\": dial tcp: address 0: missing port in address"
t=2023-06-30T17:07:00-0400 lvl=warn msg="Dqlite: attempt 3: server 0: dial: Failed connecting to HTTP endpoint \"0\": dial tcp: address 0: missing port in address"
t=2023-06-30T17:07:01-0400 lvl=warn msg="Dqlite: attempt 4: server 0: dial: Failed connecting to HTTP endpoint \"0\": dial tcp: address 0: missing port in address"
t=2023-06-30T17:07:02-0400 lvl=warn msg="Dqlite: attempt 5: server 0: dial: Failed connecting to HTTP endpoint \"0\": dial tcp: address 0: missing port in address"
t=2023-06-30T17:07:03-0400 lvl=warn msg="Dqlite: attempt 6: server 0: dial: Failed connecting to HTTP endpoint \"0\": dial tcp: address 0: missing port in address"
t=2023-06-30T17:07:04-0400 lvl=warn msg="Dqlite: attempt 7: server 0: dial: Failed connecting to HTTP endpoint \"0\": dial tcp: address 0: missing port in address"
t=2023-06-30T17:07:05-0400 lvl=warn msg="Dqlite: attempt 8: server 0: dial: Failed connecting to HTTP endpoint \"0\": dial tcp: address 0: missing port in address"
t=2023-06-30T17:07:06-0400 lvl=warn msg="Dqlite: attempt 9: server 0: dial: Failed connecting to HTTP endpoint \"0\": dial tcp: address 0: missing port in address"
t=2023-06-30T17:07:07-0400 lvl=warn msg="Dqlite: attempt 10: server 0: dial: Failed connecting to HTTP endpoint \"0\": dial tcp: address 0: missing port in address"
t=2023-06-30T17:07:08-0400 lvl=warn msg="Dqlite: attempt 11: server 0: dial: Failed connecting to HTTP endpoint \"0\": dial tcp: address 0: missing port in address"
t=2023-06-30T17:07:09-0400 lvl=warn msg="Dqlite: attempt 12: server 0: dial: Failed connecting to HTTP endpoint \"0\": dial tcp: address 0: missing port in address"
t=2023-06-30T17:07:12-0400 lvl=info msg="Connecting to global database"
t=2023-06-30T17:07:12-0400 lvl=warn msg="Dqlite: attempt 1: server 0: dial: Failed connecting to HTTP endpoint \"0\": dial tcp: address 0: missing port in address"
t=2023-06-30T17:07:12-0400 lvl=warn msg="Dqlite: attempt 2: server 0: dial: Failed connecting to HTTP endpoint \"0\": dial tcp: address 0: missing port in address"
t=2023-06-30T17:07:13-0400 lvl=warn msg="Dqlite: attempt 3: server 0: dial: Failed connecting to HTTP endpoint \"0\": dial tcp: address 0: missing port in address"
t=2023-06-30T17:07:14-0400 lvl=warn msg="Dqlite: attempt 4: server 0: dial: Failed connecting to HTTP endpoint \"0\": dial tcp: address 0: missing port in address"
t=2023-06-30T17:07:15-0400 lvl=warn msg="Dqlite: attempt 5: server 0: dial: Failed connecting to HTTP endpoint \"0\": dial tcp: address 0: missing port in address"
t=2023-06-30T17:07:16-0400 lvl=warn msg="Dqlite: attempt 6: server 0: dial: Failed connecting to HTTP endpoint \"0\": dial tcp: address 0: missing port in address"
t=2023-06-30T17:07:17-0400 lvl=warn msg="Dqlite: attempt 7: server 0: dial: Failed connecting to HTTP endpoint \"0\": dial tcp: address 0: missing port in address"
t=2023-06-30T17:07:18-0400 lvl=warn msg="Dqlite: attempt 8: server 0: dial: Failed connecting to HTTP endpoint \"0\": dial tcp: address 0: missing port in address"
t=2023-06-30T17:07:19-0400 lvl=warn msg="Dqlite: attempt 9: server 0: dial: Failed connecting to HTTP endpoint \"0\": dial tcp: address 0: missing port in address"
t=2023-06-30T17:07:20-0400 lvl=warn msg="Dqlite: attempt 10: server 0: dial: Failed connecting to HTTP endpoint \"0\": dial tcp: address 0: missing port in address"
t=2023-06-30T17:07:21-0400 lvl=warn msg="Dqlite: attempt 11: server 0: dial: Failed connecting to HTTP endpoint \"0\": dial tcp: address 0: missing port in address"
t=2023-06-30T17:07:22-0400 lvl=warn msg="Dqlite: attempt 12: server 0: dial: Failed connecting to HTTP endpoint \"0\": dial tcp: address 0: missing port in address"
t=2023-06-30T17:07:25-0400 lvl=info msg="Connecting to global database"
t=2023-06-30T17:07:25-0400 lvl=warn msg="Dqlite: attempt 1: server 0: dial: Failed connecting to HTTP endpoint \"0\": dial tcp: address 0: missing port in address"
t=2023-06-30T17:07:25-0400 lvl=warn msg="Dqlite: attempt 2: server 0: dial: Failed connecting to HTTP endpoint \"0\": dial tcp: address 0: missing port in address"
t=2023-06-30T17:07:25-0400 lvl=warn msg="Dqlite: attempt 3: server 0: dial: Failed connecting to HTTP endpoint \"0\": dial tcp: address 0: missing port in address"
t=2023-06-30T17:07:26-0400 lvl=warn msg="Dqlite: attempt 4: server 0: dial: Failed connecting to HTTP endpoint \"0\": dial tcp: address 0: missing port in address"
t=2023-06-30T17:07:27-0400 lvl=warn msg="Dqlite: attempt 5: server 0: dial: Failed connecting to HTTP endpoint \"0\": dial tcp: address 0: missing port in address"
t=2023-06-30T17:07:28-0400 lvl=warn msg="Dqlite: attempt 6: server 0: dial: Failed connecting to HTTP endpoint \"0\": dial tcp: address 0: missing port in address"
t=2023-06-30T17:07:29-0400 lvl=warn msg="Dqlite: attempt 7: server 0: dial: Failed connecting to HTTP endpoint \"0\": dial tcp: address 0: missing port in address"
t=2023-06-30T17:07:30-0400 lvl=warn msg="Dqlite: attempt 8: server 0: dial: Failed connecting to HTTP endpoint \"0\": dial tcp: address 0: missing port in address"
t=2023-06-30T17:07:31-0400 lvl=warn msg="Dqlite: attempt 9: server 0: dial: Failed connecting to HTTP endpoint \"0\": dial tcp: address 0: missing port in address"
t=2023-06-30T17:07:32-0400 lvl=warn msg="Dqlite: attempt 10: server 0: dial: Failed connecting to HTTP endpoint \"0\": dial tcp: address 0: missing port in address"
t=2023-06-30T17:07:33-0400 lvl=warn msg="Dqlite: attempt 11: server 0: dial: Failed connecting to HTTP endpoint \"0\": dial tcp: address 0: missing port in address"
t=2023-06-30T17:07:34-0400 lvl=warn msg="Dqlite: attempt 12: server 0: dial: Failed connecting to HTTP endpoint \"0\": dial tcp: address 0: missing port in address"
t=2023-06-30T17:07:37-0400 lvl=info msg="Connecting to global database"
t=2023-06-30T17:07:37-0400 lvl=warn msg="Dqlite: attempt 1: server 0: dial: Failed connecting to HTTP endpoint \"0\": dial tcp: address 0: missing port in address"
t=2023-06-30T17:07:37-0400 lvl=warn msg="Dqlite: attempt 2: server 0: dial: Failed connecting to HTTP endpoint \"0\": dial tcp: address 0: missing port in address"
t=2023-06-30T17:07:38-0400 lvl=warn msg="Dqlite: attempt 3: server 0: dial: Failed connecting to HTTP endpoint \"0\": dial tcp: address 0: missing port in address"
t=2023-06-30T17:07:38-0400 lvl=warn msg="Dqlite: attempt 4: server 0: dial: Failed connecting to HTTP endpoint \"0\": dial tcp: address 0: missing port in address"
t=2023-06-30T17:07:39-0400 lvl=warn msg="Dqlite: attempt 5: server 0: dial: Failed connecting to HTTP endpoint \"0\": dial tcp: address 0: missing port in address"
t=2023-06-30T17:07:40-0400 lvl=warn msg="Dqlite: attempt 6: server 0: dial: Failed connecting to HTTP endpoint \"0\": dial tcp: address 0: missing port in address"
t=2023-06-30T17:07:41-0400 lvl=warn msg="Dqlite: attempt 7: server 0: dial: Failed connecting to HTTP endpoint \"0\": dial tcp: address 0: missing port in address"
t=2023-06-30T17:07:42-0400 lvl=warn msg="Dqlite: attempt 8: server 0: dial: Failed connecting to HTTP endpoint \"0\": dial tcp: address 0: missing port in address"
t=2023-06-30T17:07:43-0400 lvl=warn msg="Dqlite: attempt 9: server 0: dial: Failed connecting to HTTP endpoint \"0\": dial tcp: address 0: missing port in address"
t=2023-06-30T17:07:44-0400 lvl=warn msg="Dqlite: attempt 10: server 0: dial: Failed connecting to HTTP endpoint \"0\": dial tcp: address 0: missing port in address"
t=2023-06-30T17:07:45-0400 lvl=warn msg="Dqlite: attempt 11: server 0: dial: Failed connecting to HTTP endpoint \"0\": dial tcp: address 0: missing port in address"
t=2023-06-30T17:07:46-0400 lvl=warn msg="Dqlite: attempt 12: server 0: dial: Failed connecting to HTTP endpoint \"0\": dial tcp: address 0: missing port in address"
t=2023-06-30T17:07:47-0400 lvl=eror msg="Failed connecting to global database" attempt=2115 err="failed to create dqlite connection: no available dqlite leader server found"
t=2023-06-30T17:07:49-0400 lvl=info msg="Connecting to global database"

Can you show sqlite3 /var/snap/lxd/common/lxd/database/local.db .dump?

The errors above makes it sound like LXD clustering has been enabled and is causing a bit of a mess, did you intentionally or accidentally enable LXD clustering on that system?

~$ sudo sqlite3 /var/snap/lxd/common/lxd/database/local.db .dump
[sudo] password for jeff:
PRAGMA foreign_keys=OFF;
BEGIN TRANSACTION;
CREATE TABLE config (
    id INTEGER PRIMARY KEY AUTOINCREMENT NOT NULL,
    key VARCHAR(255) NOT NULL,
    value TEXT,
    UNIQUE (key)
);
INSERT INTO config VALUES(2,'cluster.https_address','192.168.1.20:8443');
INSERT INTO config VALUES(5,'core.https_address','192.168.1.20');
CREATE TABLE patches (
    id INTEGER PRIMARY KEY AUTOINCREMENT NOT NULL,
    name VARCHAR(255) NOT NULL,
    applied_at DATETIME NOT NULL,
    UNIQUE (name)
);
INSERT INTO patches VALUES(1,'invalid_profile_names',1520761002);
INSERT INTO patches VALUES(2,'leftover_profile_config',1520761002);
INSERT INTO patches VALUES(3,'fix_uploaded_at',1520761002);
INSERT INTO patches VALUES(4,'shrink_logs_db_file',1563410920);
INSERT INTO patches VALUES(5,'network_permissions',1563410920);
INSERT INTO patches VALUES(6,'storage_api',1563410920);
INSERT INTO patches VALUES(7,'storage_api_v1',1563410920);
INSERT INTO patches VALUES(8,'storage_api_dir_cleanup',1563410920);
INSERT INTO patches VALUES(9,'storage_api_lvm_keys',1563410920);
INSERT INTO patches VALUES(10,'storage_api_keys',1563410920);
INSERT INTO patches VALUES(11,'storage_api_update_storage_configs',1563410920);
INSERT INTO patches VALUES(12,'storage_api_lxd_on_btrfs',1563410920);
INSERT INTO patches VALUES(13,'storage_api_lvm_detect_lv_size',1563410921);
INSERT INTO patches VALUES(14,'storage_api_insert_zfs_driver',1563410921);
INSERT INTO patches VALUES(15,'storage_zfs_noauto',1563410921);
INSERT INTO patches VALUES(16,'storage_zfs_volume_size',1563410921);
INSERT INTO patches VALUES(17,'network_dnsmasq_hosts',1563410921);
INSERT INTO patches VALUES(18,'storage_api_dir_bind_mount',1563410921);
INSERT INTO patches VALUES(19,'storage_api_ceph_size_remove',1563410921);
INSERT INTO patches VALUES(20,'devices_new_naming_scheme',1563410921);
INSERT INTO patches VALUES(21,'storage_api_permissions',1563410921);
INSERT INTO patches VALUES(22,'container_config_regen',1563410921);
INSERT INTO patches VALUES(23,'lvm_node_specific_config_keys',1563410921);
INSERT INTO patches VALUES(24,'candid_rename_config_key',1563410921);
INSERT INTO patches VALUES(25,'move_backups',1590893621);
INSERT INTO patches VALUES(26,'storage_api_rename_container_snapshots_dir',1590893621);
INSERT INTO patches VALUES(27,'storage_api_rename_container_snapshots_links',1590893621);
INSERT INTO patches VALUES(28,'fix_lvm_pool_volume_names',1590893621);
INSERT INTO patches VALUES(29,'storage_api_rename_container_snapshots_dir_again',1590893621);
INSERT INTO patches VALUES(30,'storage_api_rename_container_snapshots_links_again',1590893621);
INSERT INTO patches VALUES(31,'storage_api_rename_container_snapshots_dir_again_again',1590893621);
INSERT INTO patches VALUES(32,'clustering_add_roles',1590893621);
INSERT INTO patches VALUES(33,'clustering_add_roles_again',1590893621);
INSERT INTO patches VALUES(34,'storage_create_vm',1590893623);
INSERT INTO patches VALUES(35,'storage_zfs_mount',1590893623);
INSERT INTO patches VALUES(36,'network_pid_files',1590893623);
INSERT INTO patches VALUES(37,'storage_create_vm_again',1590893623);
INSERT INTO patches VALUES(38,'storage_zfs_volmode',1590893623);
INSERT INTO patches VALUES(39,'storage_rename_custom_volume_add_project',1590893623);
INSERT INTO patches VALUES(40,'storage_lvm_skipactivation',1590893623);
INSERT INTO patches VALUES(41,'clustering_drop_database_role',1593591867);
INSERT INTO patches VALUES(42,'network_clear_bridge_volatile_hwaddr',1597784375);
INSERT INTO patches VALUES(43,'network_fan_enable_nat',1604432066);
INSERT INTO patches VALUES(44,'thinpool_typo_fix',1612193179);
INSERT INTO patches VALUES(45,'vm_rename_uuid_key',1612193179);
INSERT INTO patches VALUES(46,'db_nodes_autoinc',1620309121);
INSERT INTO patches VALUES(47,'clustering_server_cert_trust',1626786707);
INSERT INTO patches VALUES(48,'dnsmasq_entries_include_device_name',1645685425);
CREATE TABLE schema (
    id         INTEGER PRIMARY KEY AUTOINCREMENT NOT NULL,
    version    INTEGER NOT NULL,
    updated_at DATETIME NOT NULL,
    UNIQUE (version)
);
INSERT INTO schema VALUES(1,32,1520761002);
INSERT INTO schema VALUES(2,33,1563410920);
INSERT INTO schema VALUES(3,34,1563410920);
INSERT INTO schema VALUES(4,35,1563410920);
INSERT INTO schema VALUES(5,36,1563410920);
INSERT INTO schema VALUES(6,37,1563410920);
INSERT INTO schema VALUES(7,38,1590893621);
INSERT INTO schema VALUES(8,39,1590893621);
INSERT INTO schema VALUES(9,40,1594330676);
INSERT INTO schema VALUES(10,41,1626786706);
INSERT INTO schema VALUES(11,42,1635811973);
CREATE TABLE raft_nodes (
    id INTEGER PRIMARY KEY AUTOINCREMENT NOT NULL,
    address TEXT NOT NULL, role INTEGER NOT NULL DEFAULT 0, name TEXT NOT NULL default "",
    UNIQUE (address)
);
INSERT INTO raft_nodes VALUES(1,'0',0,'');
CREATE TABLE certificates (
        id INTEGER PRIMARY KEY AUTOINCREMENT NOT NULL,
        fingerprint TEXT NOT NULL,
        type INTEGER NOT NULL,
        name TEXT NOT NULL,
        certificate TEXT NOT NULL,
        UNIQUE (fingerprint)
);
INSERT INTO certificates VALUES(1828,'2b5c8c5646b88fef9979ff25101937e35c0296230214e02dcb228b8c5cf5c12f',2,'cojoserver',replace('-----BEGIN CERTIFICATE-----\nMIIFcDCCA1igAwIBAgIQT79QP/DrV/MB3wGAHL3mPzANBgkqhkiG9w0BAQsFADA4\nMRwwGgYDVQQKExNsaW51eGNvbnRhaW5lcnMub3JnMRgwFgYDVQQDDA9yb290QGNv\nam9zZXJ2ZXIwHhcNMTgwMzExMDkzNjQ0WhcNMjgwMzA4MDkzNjQ0WjA4MRwwGgYD\nVQQKExNsaW51eGNvbnRhaW5lcnMub3JnMRgwFgYDVQQDDA9yb290QGNvam9zZXJ2\nZXIwggIiMA0GCSqGSIb3DQEBAQUAA4ICDwAwggIKAoICAQCy1ojWo9yse1lr1Lp7\nlZAUq5SxVZXaFJwm1aqFQ+aiY1J975Yn6uPy4EnA0/VD2nuaL8T4PI2lEwCHOsZV\n53RkjsPh99O64Q6dEUd6GvGjBFacW6jCYbN1DJx6JhEI2wzoTs6WIOs69z7UVM6i\nX0DZMA0h4TeGFMmSwTJr417/+TLf/Xj2N6NpAIBXab8Ops+bGFIlamSwFKoy0CaN\nTgjzTjrQ2QQZnPYVsSmZAJz8zPN5UUj88rkm4KVr8cErQpt6Cw5UXeRSwWR1oO8W\nGLG/EWZP8VO7oc7lDjTmxEog1Uv2dO2jJ4giT36zeyWo258OsraWPptJ4dBCubbH\n228THJhgp8s67JEVSOnBuTSip+tVOS+NcVc8D0VldUTg5YKv5GYTQDTWwAa4DF44\nwrRUv2W/Lo//ypbRnzF6ytBOdQ+9e3ZzpUh1WYDn+DlRXUCmsjRzmU8kTP2nLBvt\ntjPb9o45jtPso3MvY/3Tz7pQEihX+YR8dQ+aGziMD/k8ejKdZOsQrFGfF6r/4MyU\nrwpAyG4vxt+j+kzmdnynjJ6Rzyi82SHO09CogkylbAbABFi1Tisa/0dBr7GGf2jt\nBUjEOENuqfqQ2EwUsEqzGIgWwP8u3OvqfrkwycGcbTkDI0pwyZYcdDdiXyX0Fb5A\nONWHJm7VPluvjMwGPwZOK3eznwIDAQABo3YwdDAOBgNVHQ8BAf8EBAMCBaAwEwYD\nVR0lBAwwCgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADA/BgNVHREEODA2ggpjb2pv\nc2VydmVyhwTAqAEUhxAmAIgFMgAFz6lCoZLMudW/hxAmAIgFMgAFzxK/SP/+2BO0\nMA0GCSqGSIb3DQEBCwUAA4ICAQAnL3k9iBdli9rewSTY0+q/PcSHWiPO7DlrRx+S\niPwoZWeKh6jnugRodpxDI+3Y6ohrWLydhA7+qB+VvULZTpsJp+76ZyFwxGJL8bvS\nSrBoIrs4HS/W7lmeIIgaN4zdf3pd8ZEJpp9y3UhRq3hyyO/LzevmQ18A7GUb+QB1\n8JOU53vDUyTTPpOii1oCgkDfyobPo0Ni2u+umbJ2ESfWDvFh6L+jic6A6BYuDjzL\n4Yn+rPXt7TzG6xami18Bj0UiYEYz6UPqcLU7GD9s1TULXjeVP5e2ujQwCLn1s5uM\n0EpyDzQ2CNipjpC5IbcEK1V36pyDaU/y7F2pceqPWNzeMzkWZ+VcI2uowdyEIS1p\nokNHOMD7qONMYb1nJWXLRL0i3f1OCc5F5NrjGlOirx2TzLWM7AmkIWnXo64MvYGw\najA9YPt8s37nL3db9An+sdBVCVXbDvdl6nEFFqUp9172UQJROsK0VZiZA0V5qeXo\n8mhFEC9psDk9MTrHePc/1FWQ1+by4YWlsHm//Npmci54bA81jdvrfZtod2QB6auT\nBhHbzC3DJpOu3+TlBGd4vZcSEE91Q7XvA4KayYqeCdoS/eNTV+R3t9IOQUAsU5r/\nWuxT0eZVd2nu9h+bZ9K7ET1Fb9JhUWZq+D+13EVlfA2Sx94U3vuONefTPa6DJBkD\nFaNhRg==\n-----END CERTIFICATE-----\n','\n',char(10)));
DELETE FROM sqlite_sequence;
INSERT INTO sqlite_sequence VALUES('schema',11);
INSERT INTO sqlite_sequence VALUES('patches',48);
INSERT INTO sqlite_sequence VALUES('config',5);
INSERT INTO sqlite_sequence VALUES('raft_nodes',1);
INSERT INTO sqlite_sequence VALUES('certificates',1828);
COMMIT;

If I enabled clustering, it was definitely by accident and only during troubleshooting.

I started getting intimating connection to my file server a few weeks back.
I did a zpool scrub to make sure my file system hardware was ok.
No errors reported

Two weeks ago, I created a container to host wordpress.
Not sure if that broke anything.

I rebooted the server and lost all containers. All lxc or lxd command hang indefinitely.

Definitely a system which was set to be clustered, but looks like something went wrong and it got kinda stuck in between standalone and clustered mode.

The config shows that it’s configured to run at 192.168.1.20 for the cluster but then the database shows an empty address.

Unfortunately you’re on 4.0 which is a bit old at this point and doesn’t have all the nice recovery commands we have in 5.0 and higher. I’ll setup a VM with 4.0 to look at what the options were back then.

Good afternoon Stéphane,

Greatly appreciate your assistance with the recovery and the work you are doing with the LXD project. Hopefully we can get these containers back online and I can update to the current version.

Patiently standing by…

You can try snap stop lxd, if that hangs, you may need to also run kill -9 $(cat /var/snap/lxd/common/lxd.pid) to unstick it.

Once LXD is fully stopped, run lxd cluster recover-from-quorum-loss, if that succeeds, then try snap start lxd and we’ll see if that sorts it out.

If not, you’ll likely need to stop LXD again and this time use lxd cluster edit to reconfigure the cluster with a valid address (192.168.1.20:8443) and see if that brings the database back up.

snap stop lxd hangs

lxd cluster recover-from-quorum-loss
Error: This LXD instance has no database role

lxd cluster edit

# Latest dqlite segment ID: 9765994

members:
- id: 1
  address: "192.168.1.20:8443"
  role: voter

Config validation error: This cluster member has no raft role
Press enter to open the editor again or ctrl+c to abort change

Looks like it’s looking for id: 0 in the logs, maybe use lxd cluster edit to see if you can’t change the id to 0?

unfortunately, the error message changes to this;

Config validation error: Changing cluster member ID is not supported
Press enter to open the editor again or ctrl+c to abort change

Can you send me a tarball of /var/snap/lxd/common/lxd/database to stgraber@ubuntu.com?

That should let me load your database in a VM here and see if I can recover it or at worse, I’ll just extract the data from it and dump it into a clean (non-clustered) database.

1 Like

Greeting,

Tarball of database emailed to you. Thanks in advance.

Got it, I’ll try to look at it later tonight.

Thank you