Hi, I was hoping somebody could help me, as I’m at a loss I’ve not been able to use LXC on my main server, I think due to the second server been dead, due to 3 drive’s failing on its raid 6 array, it’s non-recoverable. My main server now fails to link to the cluster as there is no node top connect to, all the ZFS container data is on main sever that still live and assessable through ZFS mount tools but I can’t get LXC to run them or for that matter even start any command due to connection issue, this is only my assume but do not have access to the command-line tool to disconnect cluster I’ve removed LXD and re-installed .deb again with apt.
#Here is the outputs I am getting
sudo lxc list
Error: Get http://unix.socket/1.0: EOF
sudo lxc profile show default
Error: Get http://unix.socket/1.0: EOF
sudo lxd cluster list-database --debug
DBUG[05-18|13:17:13] Connecting to a local LXD over a Unix socket
DBUG[05-18|13:17:13] Sending request to LXD method=GET url=http://unix.socket/1.0 etag=
journalctl -u lxd -n 300
May 18 14:34:56 zenicmain1 lxd[28758]: t=2020-05-18T14:34:56+0100 lvl=warn msg=“Raft: Election timeout reached, restarting election”
May 18 14:34:57 zenicmain1 lxd[28758]: t=2020-05-18T14:34:57+0100 lvl=warn msg="Failed connecting to global database (attempt 10): failed to create dqlite
sudo systemctl status lxd.socket
● lxd.socket - LXD - unix socket
Loaded: loaded (/lib/systemd/system/lxd.socket; enabled; vendor preset: enabled)
Active: active (running) since Mon 2020-05-18 12:19:37 BST; 7min ago
Docs: man:lxd(1)
Listen: /var/lib/lxd/unix.socket (Stream)
Tasks: 0 (limit: 4915)
CGroup: /system.slice/lxd.socket
sudo systemctl status lxd
● lxd.service - LXD - main daemon
Loaded: loaded (/lib/systemd/system/lxd.service; indirect; vendor preset: enabled)
Active: activating (start-post) since Mon 2020-05-18 15:02:39 BST; 6min ago
Docs: man:lxd(1)
Process: 29067 ExecStartPre=/usr/lib/x86_64-linux-gnu/lxc/lxc-apparmor-load (code=exited, status=0/SUCCESS)
Main PID: 29083 (lxd); Control PID: 29084 (lxd)
Tasks: 40
CGroup: /system.slice/lxd.service
├─29083 /usr/lib/lxd/lxd --group lxd --logfile=/var/log/lxd/lxd.log
└─29084 /usr/lib/lxd/lxd waitready --timeout=600
May 18 15:08:33 zenicmain1 lxd[29083]: t=2020-05-18T15:08:33+0100 lvl=warn msg=“Raft: Election timeout reached, restarting election”
May 18 15:08:38 zenicmain1 lxd[29083]: t=2020-05-18T15:08:38+0100 lvl=warn msg=“Raft: Election timeout reached, restarting election”
May 18 15:08:44 zenicmain1 lxd[29083]: t=2020-05-18T15:08:44+0100 lvl=warn msg=“Raft: Election timeout reached, restarting election”
May 18 15:08:50 zenicmain1 lxd[29083]: t=2020-05-18T15:08:50+0100 lvl=warn msg=“Raft: Election timeout reached, restarting election”
May 18 15:08:54 zenicmain1 lxd[29083]: t=2020-05-18T15:08:54+0100 lvl=warn msg=“Raft: Election timeout reached, restarting election”
May 18 15:08:59 zenicmain1 lxd[29083]: t=2020-05-18T15:08:59+0100 lvl=warn msg=“Raft: Election timeout reached, restarting election”
May 18 15:09:02 zenicmain1 lxd[29083]: t=2020-05-18T15:09:02+0100 lvl=warn msg="Failed connecting to global database (attempt 30): failed to create dqlite
May 18 15:09:02 zenicmain1 lxd[29083]: t=2020-05-18T15:09:02+0100 lvl=warn msg=“Raft: Election timeout reached, restarting election”
May 18 15:09:06 zenicmain1 lxd[29083]: t=2020-05-18T15:09:06+0100 lvl=warn msg=“Raft: Election timeout reached, restarting election”
May 18 15:09:10 zenicmain1 lxd[29083]: t=2020-05-18T15:09:10+0100 lvl=warn msg=“Raft: Election timeout reached, restarting election”
sudo systemctl status lxc
● lxc.service - LXC Container Initialization and Autoboot Code
Loaded: loaded (/lib/systemd/system/lxc.service; enabled; vendor preset: enabled)
Active: active (exited) since Sat 2020-05-16 23:51:54 BST; 1 day 15h ago
Docs: man:lxc-autostart
man:lxc
Process: 2736 ExecStart=/usr/lib/x86_64-linux-gnu/lxc/lxc-containers start (code=exited, status=0/SUCCESS)
Process: 2719 ExecStartPre=/usr/lib/x86_64-linux-gnu/lxc/lxc-apparmor-load (code=exited, status=0/SUCCESS)
Main PID: 2736 (code=exited, status=0/SUCCESS)
May 16 23:51:53 zenicmain1 systemd[1]: Starting LXC Container Initialization and Autoboot Code…
May 16 23:51:54 zenicmain1 systemd[1]: Started LXC Container Initialization and Autoboot Code.
ps aux | grep -i lxd
zenic 27328 0.0 0.0 14428 1104 pts/0 S+ 12:22 0:00 grep --color=auto -i lxd
#Here is things I’ve tried
ps aux | grep -i lxc
root 1974 0.0 0.0 235764 2148 ? Ssl May16 0:01 /usr/bin/lxcfs /var/lib/lxcfs/
lxc-dns+ 2714 0.0 0.0 52880 376 ? S May16 0:00 dnsmasq -u lxc-dnsmasq --strict-order --bind-interfaces --pid-file=/run/lxc/dnsmasq.pid --listen-address 10.0.2.1 --dhcp-range 10.0.2.2,10.0.2.254 --dhcp-lease-max=253 --dhcp-no-override --except-interface=lo --interface=lxcbr0 --dhcp-leasefile=/var/lib/misc/dnsmasq.lxcbr0.leases --dhcp-authoritative
root 28097 0.0 0.0 65612 4344 pts/0 T 13:24 0:00 sudo lxc config set core.https_address [::]:8443
root 28098 0.0 0.0 396516 12848 pts/0 Tl 13:24 0:00 lxc config set core.https_address [::]:8443
root 28121 0.0 0.0 65612 4216 pts/0 T 13:25 0:00 sudo lxc config show
root 28122 0.0 0.0 394852 12268 pts/0 Tl 13:25 0:00 lxc config show
zenic 29347 0.0 0.0 14428 1136 pts/0 S+ 15:27 0:00 grep --color=auto -i lxc
sudo systemctl reload lxc
Failed to reload lxc.service: Job type reload is not applicable for unit lxc.service.
See system logs and ‘systemctl status lxc.service’ for details.
systemctl status lxc.service
● lxc.service - LXC Container Initialization and Autoboot Code
Loaded: loaded (/lib/systemd/system/lxc.service; enabled; vendor preset: enabled)
Active: active (exited) since Sat 2020-05-16 23:51:54 BST; 1 day 16h ago
Docs: man:lxc-autostart
man:lxc
Process: 2736 ExecStart=/usr/lib/x86_64-linux-gnu/lxc/lxc-containers start (code=exited, status=0/SUCCESS)
Process: 2719 ExecStartPre=/usr/lib/x86_64-linux-gnu/lxc/lxc-apparmor-load (code=exited, status=0/SUCCESS)
Main PID: 2736 (code=exited, status=0/SUCCESS)
May 16 23:51:53 zenicmain1 systemd[1]: Starting LXC Container Initialization and Autoboot Code…
May 16 23:51:54 zenicmain1 systemd[1]: Started LXC Container Initialization and Autoboot Code.
sudo systemctl status lxd-containers.service
● lxd-containers.service - LXD - container startup/shutdown
Loaded: loaded (/lib/systemd/system/lxd-containers.service; enabled; vendor preset: enabled)
Active: failed (Result: timeout) since Sun 2020-05-17 00:01:46 BST; 1 day 17h ago
Docs: man:lxd(1)
Process: 1919 ExecStart=/usr/bin/lxd activateifneeded (code=killed, signal=TERM)
Main PID: 1919 (code=killed, signal=TERM)
May 16 23:51:46 zenicmain1 systemd[1]: Starting LXD - container startup/shutdown…
May 17 00:01:46 zenicmain1 systemd[1]: lxd-containers.service: Start operation timed out. Terminating.
May 17 00:01:46 zenicmain1 systemd[1]: lxd-containers.service: Main process exited, code=killed, status=15/TERM
May 17 00:01:46 zenicmain1 systemd[1]: lxd-containers.service: Failed with result ‘timeout’.
May 17 00:01:46 zenicmain1 systemd[1]: Failed to start LXD - container startup/shutdown.
systemctl stop lxc.service
systemctl stop lxd.service
systemctl stop lxd.socket
LXD Hangs on restart
systemctl start lxd.service
LXD socket did need to be restarted as systemctl start lxd.service started it even though it hangs
systemctl status lxd.socket
● lxd.socket - LXD - unix socket
Loaded: loaded (/lib/systemd/system/lxd.socket; enabled; vendor preset: enabled)
Active: active (running) since Mon 2020-05-18 16:57:38 BST; 5min ago
Docs: man:lxd(1)
Listen: /var/lib/lxd/unix.socket (Stream)
Tasks: 0 (limit: 4915)
CGroup: /system.slice/lxd.socket
May 18 16:57:38 zenicmain1 systemd[1]: Starting LXD - unix socket.
May 18 16:57:38 zenicmain1 systemd[1]: Listening on LXD - unix socket.
LXC starts fine
systemctl status lxc.service
● lxc.service - LXC Container Initialization and Autoboot Code
Loaded: loaded (/lib/systemd/system/lxc.service; enabled; vendor preset: enabled)
Active: active (exited) since Mon 2020-05-18 17:01:23 BST; 19s ago
Docs: man:lxc-autostart
man:lxc
Process: 30223 ExecStop=/usr/lib/x86_64-linux-gnu/lxc/lxc-containers stop (code=exited, status=1/FAILURE)
Process: 30436 ExecStart=/usr/lib/x86_64-linux-gnu/lxc/lxc-containers start (code=exited, status=0/SUCCESS)
Process: 30419 ExecStartPre=/usr/lib/x86_64-linux-gnu/lxc/lxc-apparmor-load (code=exited, status=0/SUCCESS)
Main PID: 30436 (code=exited, status=0/SUCCESS)
May 18 17:01:23 zenicmain1 systemd[1]: Starting LXC Container Initialization and Autoboot Code…
May 18 17:01:23 zenicmain1 systemd[1]: Started LXC Container Initialization and Autoboot Code.
sudo lxc config set core.https_address [::]:8443 --debug
DBUG[05-18|17:19:21] Connecting to a local LXD over a Unix socket
DBUG[05-18|17:19:21] Sending request to LXD method=GET url=http://unix.socket/1.0 etag=
Error: Get http://unix.socket/1.0: EOF
just hangs until I get the error message
Still can’t get LXC to work
#Here is things am looking at next but really could do with some advice/help
https://github.com/lxc/lxd/issues/5802
https://lxd.readthedocs.io/en/latest/database/
I’ve also restarted all the service for LXD LXC, I’ve tried killing LXD processes but doesn’t seem to stop even when LXD is stoped, I think I am looking to edit a .yaml or SQLite database to remove connection as I cant do it from LXC command line any help would be greatly appreciated many thanks ben