After running for a year now without an issue it has stopped working. I could no longer do lxc list.
I reboot 3 out of the 4 servers, this usually got it working. One server is still serving web pages and I don’t want to risk this by rebooting it. All the servers are running lxd fine, just I get Error: Get “http://unix.socket/1.0”: EOF when I do lxc list. Some 2 show lxd version 5.2 , 2 other show : 5.3
Any ideas on how to get this party going…
Thanks is advance
What does /var/snap/lxd/common/lxd/logs/lxd.log on the 3 problem servers show?
Q1 says… time=“2022-07-15T20:24:39-04:00” level=warning msg=“Wait for other cluster nodes to upgrade their versions, cluster not started yet”
Q2 says… time=“2022-07-15T20:25:51-04:00” level=warning msg=“Cluster member info not found” address=“84.17.40.59:8443”
time=“2022-07-15T20:25:51-04:00” level=debug msg=“Starting heartbeat round” local=“84.17.40.19:8443” mode=normal
time=“2022-07-15T20:25:51-04:00” level=debug msg=“Heartbeat updating local raft members” members="[{{1 84.17.40.18:8443 voter} Q1} {{2 84.17.40.20:8443 spare} Q3} {{3 84.17.40.19:8443 voter} Q2} {{4 84.17.40.21:8443 voter} Q4} {{5 84.17.40.59:8443 spare} }]"
time=“2022-07-15T20:25:51-04:00” level=error msg=“Unaccounted raft node(s) not found in ‘nodes’ table for heartbeat: {NodeInfo:{ID:5 Address:84.17.40.59:8443 Role:spare} Name:}”
time=“2022-07-15T20:25:51-04:00” level=debug msg=“Matched trusted cert” fingerprint=25d36dc2d614d50d9b4f477363c7f35ad7b0d06fee1b327d65c15ae41999f3be subject=“CN=root@q4,O=linuxcontainers.org”
time=“2022-07-15T20:25:51-04:00” level=info msg=“Dqlite proxy started” local=“84.17.40.19:8443” name=dqlite remote=“84.17.40.21:54388”
time=“2022-07-15T20:25:51-04:00” level=debug msg=“Error Response\n\t{\n\t\t"type”: “error”,\n\t\t"status": “”,\n\t\t"status_code": 0,\n\t\t"operation": “”,\n\t\t"error_code": 503,\n\t\t"error": “LXD daemon setup in progress”,\n\t\t"metadata": null\n\t}" http_code=503
time=“2022-07-15T20:25:51-04:00” level=info msg=“Dqlite proxy stopped” local=“84.17.40.19:36908” name=raft remote=“84.17.40.21:8443”
time=“2022-07-15T20:25:51-04:00” level=info msg=“Dqlite proxy stopped” local=“84.17.40.19:8443” name=dqlite remote=“84.17.40.21:54388”
time=“2022-07-15T20:25:51-04:00” level=info msg=“Dqlite proxy stopped” local=“84.17.40.19:8443” name=dqlite remote=“84.17.40.21:54386”
time=“2022-07-15T20:25:52-04:00” level=debug msg=“Matched trusted cert” fingerprint=25d36dc2d614d50d9b4f477363c7f35ad7b0d06fee1b327d65c15ae41999f3be subject=“CN=root@q4,O=linuxcontainers.org”
time=“2022-07-15T20:25:52-04:00” level=info msg=“Dqlite proxy started” local=“84.17.40.19:8443” name=dqlite remote=“84.17.40.21:54394”
time=“2022-07-15T20:25:53-04:00” level=info msg=“Dqlite connected outbound” local=“84.17.40.19:36914” name=raft remote=“84.17.40.21:8443”
time=“2022-07-15T20:25:53-04:00” level=info msg=“Dqlite proxy started” local=“84.17.40.19:36914” name=raft remote=“84.17.40.21:8443”
time=“2022-07-15T20:25:53-04:00” level=debug msg=“Matched trusted cert” fingerprint=25d36dc2d614d50d9b4f477363c7f35ad7b0d06fee1b327d65c15ae41999f3be subject=“CN=root@q4,O=linuxcontainers.org”
time=“2022-07-15T20:25:53-04:00” level=info msg=“Dqlite proxy started” local=“84.17.40.19:8443” name=dqlite remote=“84.17.40.21:54396
Q3 says… time=“2022-07-14T22:32:55-04:00” level=debug msg=“Initializing database gateway”
time=“2022-07-14T22:32:55-04:00” level=info msg=“Starting database node” id=2 local=“84.17.40.20:8443” role=spare
time=“2022-07-14T22:32:55-04:00” level=info msg=“Starting cluster handler:”
time=“2022-07-14T22:32:55-04:00” level=info msg=“Starting /dev/lxd handler:”
time=“2022-07-14T22:32:55-04:00” level=info msg=” - binding devlxd socket" socket=/var/snap/lxd/common/lxd/devlxd/sock
time=“2022-07-14T22:32:55-04:00” level=info msg=“REST API daemon:”
time=“2022-07-14T22:32:55-04:00” level=info msg=" - binding Unix socket" inherited=true socket=/var/snap/lxd/common/lxd/unix.socket
time=“2022-07-14T22:32:55-04:00” level=info msg=" - binding TCP socket" socket=“84.17.40.20:8443”
time=“2022-07-14T22:32:55-04:00” level=info msg=“Initializing global database”
time=“2022-07-14T22:32:55-04:00” level=info msg=“Connecting to global database”
time=“2022-07-14T22:32:55-04:00” level=info msg=“Dqlite connected outbound” local=“84.17.40.20:36178” name=dqlite remote=“84.17.40.18:8443”
time=“2022-07-14T22:32:55-04:00” level=debug msg=“Dqlite: attempt 1: server 84.17.40.18:8443: connect to reported leader 84.17.40.19:8443”
time=“2022-07-14T22:32:55-04:00” level=info msg=“Dqlite connected outbound” local=“84.17.40.20:60032” name=dqlite remote=“84.17.40.19:8443”
time=“2022-07-14T22:32:55-04:00” level=debug msg=“Dqlite: attempt 1: server 84.17.40.18:8443: connected”
time=“2022-07-14T22:32:55-04:00” level=info msg=“Connected to global database”
time=“2022-07-14T22:32:55-04:00” level=debug msg=“Database error” err=“this node’s version is behind, please upgrade”
time=“2022-07-14T22:32:55-04:00” level=error msg=“Failed to start the daemon” err=“Failed to initialize global database: failed to ensure schema: this node’s version is behind, please upgrade”
time=“2022-07-14T22:32:55-04:00” level=info msg=“Starting shutdown sequence” signal=interrupt
time=“2022-07-14T22:32:55-04:00” level=info msg=“Handing over cluster member role” address=“84.17.40.20:8443”
time=“2022-07-14T22:32:55-04:00” level=debug msg=“Connecting to a remote LXD over HTTPS”
time=“2022-07-14T22:32:55-04:00” level=debug msg=“Sending request to LXD” etag= method=POST url=“https://84.17.40.19:8443/internal/cluster/handover”
time=“2022-07-14T22:32:55-04:00” level=debug msg="\n\t{\n\t\t"address": “84.17.40.20:8443”\n\t}"
time=“2022-07-14T22:32:55-04:00” level=warning msg=“Could not handover member’s responsibilities” err=“LXD daemon setup in progress”
time=“2022-07-14T22:32:55-04:00” level=debug msg=“Cancel ongoing or future gRPC connection attempts”
time=“2022-07-14T22:32:55-04:00” level=debug msg=“Cancel ongoing or future gRPC connection attempts”
time=“2022-07-14T22:32:55-04:00” level=info msg=“Stop database gateway”
time=“2022-07-14T22:32:55-04:00” level=info msg=“Stopping REST API handler:”
time=“2022-07-14T22:32:55-04:00” level=info msg=" - closing socket" socket=“84.17.40.20:8443”
time=“2022-07-14T22:32:55-04:00” level=info msg=" - closing socket" socket=/var/snap/lxd/common/lxd/unix.socket
time=“2022-07-14T22:32:55-04:00” level=info msg=“Stopping /dev/lxd handler:”
time=“2022-07-14T22:32:55-04:00” level=info msg=" - closing socket" socket=/var/snap/lxd/common/lxd/devlxd/sock
time=“2022-07-14T22:32:55-04:00” level=info msg=“Not unmounting temporary filesystems (instances are still running)”
time=“2022-07-14T22:32:55-04:00” level=info msg=“Daemon stopped”
Q4 says… time=“2022-07-15T20:25:45-04:00” level=warning msg=" - Couldn’t find the CGroup memory swap accounting, swap limits will be ignored"
time=“2022-07-15T20:25:46-04:00” level=error msg=“Failed to start the daemon” err=“Failed to initialize global database: failed to ensure schema: this node’s version is behind, please upgrade”
time=“2022-07-15T20:25:47-04:00” level=warning msg=“Could not handover member’s responsibilities” err=“LXD daemon setup in progress”
time=“2022-07-15T20:25:48-04:00” level=warning msg=“Dqlite proxy failed” err=“first: local → remote: read unix @->@0c0dc: read: connection reset by peer” local=“84.17.40.21:8443” name=dqlite remote=“84.17.40.19:36902”
So two are complaining about the upgrade status of others… Going to try to upgrade them somehow.
Q1 says
sqlite3 /var/snap/lxd/common/lxd/database/global/db.bin “SELECT * FROM nodes;”
1|Q1||84.17.40.18:8443|60|312|2022-07-04 10:01:32.462403214-04:00|0|2|
2|Q3||84.17.40.20:8443|60|313|2022-07-04 10:06:32.382257763-04:00|0|2|
3|Q2||84.17.40.19:8443|60|315|2022-07-04 10:06:31.618628733-04:00|0|2|
4|Q4||84.17.40.21:8443|60|313|2022-07-04 10:06:34.475008997-04:00|0|2|
Q2 says
sqlite3 /var/snap/lxd/common/lxd/database/global/db.bin “SELECT * FROM nodes;”
1|Q1||84.17.40.18:8443|60|312|2022-07-04 10:01:32.462403214-04:00|0|2|
2|Q3||84.17.40.20:8443|60|313|2022-07-04 10:06:32.382257763-04:00|0|2|
3|Q2||84.17.40.19:8443|60|315|2022-07-04 10:06:31.618628733-04:00|0|2|
4|Q4||84.17.40.21:8443|60|313|2022-07-04 10:06:34.475008997-04:00|0|2|
Q3 Says
1|Q1||84.17.40.18:8443|60|306|2022-05-28 11:00:49.07030894-04:00|0|2|
2|Q3||84.17.40.20:8443|60|306|2022-05-28 11:00:47.088539154-04:00|0|2|
3|Q2||84.17.40.19:8443|60|306|2022-05-28 11:00:39.099070585-04:00|0|2|
4|Q4||84.17.40.21:8443|60|306|2022-05-28 11:00:48.794241424-04:00|0|2|
Q4 says
1|Q1||84.17.40.18:8443|60|315|2022-07-15 20:40:45.916359296-04:00|0|2|
2|Q3||84.17.40.20:8443|60|313|2022-07-04 10:08:02.294602249-04:00|0|2|
3|Q2||84.17.40.19:8443|60|315|2022-07-15 20:40:41.736339108-04:00|0|2|
4|Q4||84.17.40.21:8443|60|313|2022-07-15 20:38:02.471201464-04:00|0|2|
Can you show ‘snap info lxd’ on each server please
Q1snap info lxd
name: lxd
summary: LXD - container and VM manager
publisher: Canonical✓
store-url: Install lxd on Linux | Snap Store
contact: Issues · lxc/lxd · GitHub
license: unset
description: |
LXD is a system container and virtual machine manager.
It offers a simple CLI and REST API to manage local or remote instances,
uses an image based workflow and support for a variety of advanced features.
Images are available for all Ubuntu releases and architectures as well
as for a wide number of other Linux distributions. Existing
integrations with many deployment and operation tools, makes it work
just like a public cloud, except everything is under your control.
LXD containers are lightweight, secure by default and a great
alternative to virtual machines when running Linux on Linux.
LXD virtual machines are modern and secure, using UEFI and secure-boot
by default and a great choice when a different kernel or operating
system is needed.
With clustering, up to 50 LXD servers can be easily joined and managed
together with the same tools and APIs and without needing any external
dependencies.
Supported configuration options for the snap (snap set lxd [=…]):
- ceph.builtin: Use snap-specific Ceph configuration [default=false]
- ceph.external: Use the system's ceph tools (ignores ceph.builtin) [default=false]
- criu.enable: Enable experimental live-migration support [default=false]
- daemon.debug: Increase logging to debug level [default=false]
- daemon.group: Set group of users that have full control over LXD [default=lxd]
- daemon.user.group: Set group of users that have restricted LXD access [default=lxd]
- daemon.preseed: Pass a YAML configuration to `lxd init` on initial start
- daemon.syslog: Send LXD log events to syslog [default=false]
- daemon.verbose: Increase logging to verbose level [default=false]
- lvm.external: Use the system's LVM tools [default=false]
- lxcfs.pidfd: Start per-container process tracking [default=false]
- lxcfs.loadavg: Start tracking per-container load average [default=false]
- lxcfs.cfs: Consider CPU shares for CPU usage [default=false]
- openvswitch.builtin: Run a snap-specific OVS daemon [default=false]
- openvswitch.external: Use the system's OVS tools (ignores openvswitch.builtin) [default=false]
- ovn.builtin: Use snap-specific OVN configuration [default=false]
- shiftfs.enable: Enable shiftfs support [default=auto]
For system-wide configuration of the CLI, place your configuration in
/var/snap/lxd/common/global-conf/ (config.yml and servercerts)
commands:
Q2
snap info lxd
name: lxd
summary: LXD - container and VM manager
publisher: Canonical✓
store-url: Install lxd on Linux | Snap Store
contact: Issues · lxc/lxd · GitHub
license: unset
description: |
LXD is a system container and virtual machine manager.
It offers a simple CLI and REST API to manage local or remote instances,
uses an image based workflow and support for a variety of advanced features.
Images are available for all Ubuntu releases and architectures as well
as for a wide number of other Linux distributions. Existing
integrations with many deployment and operation tools, makes it work
just like a public cloud, except everything is under your control.
LXD containers are lightweight, secure by default and a great
alternative to virtual machines when running Linux on Linux.
LXD virtual machines are modern and secure, using UEFI and secure-boot
by default and a great choice when a different kernel or operating
system is needed.
With clustering, up to 50 LXD servers can be easily joined and managed
together with the same tools and APIs and without needing any external
dependencies.
Supported configuration options for the snap (snap set lxd [=…]):
- ceph.builtin: Use snap-specific Ceph configuration [default=false]
- ceph.external: Use the system's ceph tools (ignores ceph.builtin) [default=false]
- criu.enable: Enable experimental live-migration support [default=false]
- daemon.debug: Increase logging to debug level [default=false]
- daemon.group: Set group of users that have full control over LXD [default=lxd]
- daemon.user.group: Set group of users that have restricted LXD access [default=lxd]
- daemon.preseed: Pass a YAML configuration to `lxd init` on initial start
- daemon.syslog: Send LXD log events to syslog [default=false]
- daemon.verbose: Increase logging to verbose level [default=false]
- lvm.external: Use the system's LVM tools [default=false]
- lxcfs.pidfd: Start per-container process tracking [default=false]
- lxcfs.loadavg: Start tracking per-container load average [default=false]
- lxcfs.cfs: Consider CPU shares for CPU usage [default=false]
- openvswitch.builtin: Run a snap-specific OVS daemon [default=false]
- openvswitch.external: Use the system's OVS tools (ignores openvswitch.builtin) [default=false]
- ovn.builtin: Use snap-specific OVN configuration [default=false]
- shiftfs.enable: Enable shiftfs support [default=auto]
For system-wide configuration of the CLI, place your configuration in
/var/snap/lxd/common/global-conf/ (config.yml and servercerts)
commands:
Q3
snap info lxd
name: lxd
summary: LXD - container and VM manager
publisher: Canonical✓
store-url: Install lxd on Linux | Snap Store
contact: Issues · lxc/lxd · GitHub
license: unset
description: |
LXD is a system container and virtual machine manager.
It offers a simple CLI and REST API to manage local or remote instances,
uses an image based workflow and support for a variety of advanced features.
Images are available for all Ubuntu releases and architectures as well
as for a wide number of other Linux distributions. Existing
integrations with many deployment and operation tools, makes it work
just like a public cloud, except everything is under your control.
LXD containers are lightweight, secure by default and a great
alternative to virtual machines when running Linux on Linux.
LXD virtual machines are modern and secure, using UEFI and secure-boot
by default and a great choice when a different kernel or operating
system is needed.
With clustering, up to 50 LXD servers can be easily joined and managed
together with the same tools and APIs and without needing any external
dependencies.
Supported configuration options for the snap (snap set lxd [=…]):
- ceph.builtin: Use snap-specific Ceph configuration [default=false]
- ceph.external: Use the system's ceph tools (ignores ceph.builtin) [default=false]
- criu.enable: Enable experimental live-migration support [default=false]
- daemon.debug: Increase logging to debug level [default=false]
- daemon.group: Set group of users that have full control over LXD [default=lxd]
- daemon.user.group: Set group of users that have restricted LXD access [default=lxd]
- daemon.preseed: Pass a YAML configuration to `lxd init` on initial start
- daemon.syslog: Send LXD log events to syslog [default=false]
- lvm.external: Use the system's LVM tools [default=false]
- lxcfs.pidfd: Start per-container process tracking [default=false]
- lxcfs.loadavg: Start tracking per-container load average [default=false]
- lxcfs.cfs: Consider CPU shares for CPU usage [default=false]
- openvswitch.builtin: Run a snap-specific OVS daemon [default=false]
- openvswitch.external: Use the system's OVS tools (ignores openvswitch.builtin) [default=false]
- ovn.builtin: Use snap-specific OVN configuration [default=false]
- shiftfs.enable: Enable shiftfs support [default=auto]
For system-wide configuration of the CLI, place your configuration in
/var/snap/lxd/common/global-conf/ (config.yml and servercerts)
commands:
Q4
snap info lxd
name: lxd
summary: LXD - container and VM manager
publisher: Canonical✓
store-url: Install lxd on Linux | Snap Store
contact: Issues · lxc/lxd · GitHub
license: unset
description: |
LXD is a system container and virtual machine manager.
It offers a simple CLI and REST API to manage local or remote instances,
uses an image based workflow and support for a variety of advanced features.
Images are available for all Ubuntu releases and architectures as well
as for a wide number of other Linux distributions. Existing
integrations with many deployment and operation tools, makes it work
just like a public cloud, except everything is under your control.
LXD containers are lightweight, secure by default and a great
alternative to virtual machines when running Linux on Linux.
LXD virtual machines are modern and secure, using UEFI and secure-boot
by default and a great choice when a different kernel or operating
system is needed.
With clustering, up to 50 LXD servers can be easily joined and managed
together with the same tools and APIs and without needing any external
dependencies.
Supported configuration options for the snap (snap set lxd [=…]):
- ceph.builtin: Use snap-specific Ceph configuration [default=false]
- ceph.external: Use the system's ceph tools (ignores ceph.builtin) [default=false]
- criu.enable: Enable experimental live-migration support [default=false]
- daemon.debug: Increase logging to debug level [default=false]
- daemon.group: Set group of users that have full control over LXD [default=lxd]
- daemon.user.group: Set group of users that have restricted LXD access [default=lxd]
- daemon.preseed: Pass a YAML configuration to `lxd init` on initial start
- daemon.syslog: Send LXD log events to syslog [default=false]
- lvm.external: Use the system's LVM tools [default=false]
- lxcfs.pidfd: Start per-container process tracking [default=false]
- lxcfs.loadavg: Start tracking per-container load average [default=false]
- lxcfs.cfs: Consider CPU shares for CPU usage [default=false]
- openvswitch.builtin: Run a snap-specific OVS daemon [default=false]
- openvswitch.external: Use the system's OVS tools (ignores openvswitch.builtin) [default=false]
- ovn.builtin: Use snap-specific OVN configuration [default=false]
- shiftfs.enable: Enable shiftfs support [default=auto]
For system-wide configuration of the CLI, place your configuration in
/var/snap/lxd/common/global-conf/ (config.yml and servercerts)
commands:
Have you run snap refresh lxd on q3 and q4 as they are behind.
snap refresh lxd on all of them seem to fix the problem. Thanks