None of the lxc commands working after apt upgrade on ubuntu bionic

Problem:
No response to “lxc commands” on any of the node after doing apt update on the distribution.

Also noticed there are more then 100,000 connections from other two nodes to master node. I just used part of the connection in netstat -an command below.

Steps to reproduce

  1. Run apt upgrade on nodes in a cluster.
  2. Try to run lxc list on any node in the cluster.

Output of lxc info in debug mode

DBUG[08-17|04:50:20] Connecting to a local LXD over a Unix socket
DBUG[08-17|04:50:20] Sending request to LXD                   etag= method=GET url=http://unix.socket/1.0

LXC Info
This was done previously before update:

api_version: "1.0"
auth: trusted
public: false
auth_methods:
- tls
environment:
  addresses:
  - 10.8.8.2:8443
  architectures:
  - x86_64
  - i686
  certificate:
    -----BEGIN CERTIFICATE-----
    -----END CERTIFICATE-----
  certificate_fingerprint:
driver: lxc
  driver_version: 3.0.1
  kernel: Linux
  kernel_architecture: x86_64
  kernel_version: 4.15.0-1014-gcp
  server: lxd
  server_pid: 29049
  server_version: "3.3"
  storage: btrfs
storage_version: "4.4"
  server_clustered: true
  server_name: test

Additional information when trying to stop the lxd daemon using snap:

sudo snap stop lxd
error: cannot perform the following tasks:
- stop of [lxd.daemon] (# systemctl stop snap.lxd.daemon.service

<exceeded maximum runtime of 1m1s>)
- stop of [lxd.daemon] (exceeded maximum runtime of 1m1s)

netstat -an

tcp        0      0 10.8.8.2:8443           0.0.0.0:*               LISTEN
tcp        0      0 10.8.8.2:8443           10.8.8.4:58631          ESTABLISHED
tcp        0      0 10.8.8.2:8443           10.8.8.3:35131          ESTABLISHED
tcp        0      0 10.8.8.2:8443           10.8.8.4:43121          ESTABLISHED
tcp        0      0 10.8.8.2:8443           10.8.8.3:58260          ESTABLISHED
tcp        0      0 10.8.8.2:8443           10.8.8.4:53067          ESTABLISHED
tcp        0      0 10.8.8.2:8443           10.8.8.4:45395          ESTABLISHED
tcp        0      0 10.8.8.2:8443           10.8.8.4:39579          ESTABLISHED
tcp        0      0 10.8.8.2:8443           10.8.8.4:45221          ESTABLISHED
tcp        0      0 10.8.8.2:8443           10.8.8.4:57022          ESTABLISHED
tcp        0      0 10.8.8.2:8443           10.8.8.4:43262          ESTABLISHED
tcp        0      0 10.8.8.2:8443           10.8.8.4:42859          ESTABLISHED
tcp        0      0 10.8.8.2:8443           10.8.8.3:37665          ESTABLISHED
tcp        0      0 10.8.8.2:8443           10.8.8.4:37404          ESTABLISHED
tcp        0      0 10.8.8.2:8443           10.8.8.3:50994          ESTABLISHED
tcp        0      0 10.8.8.2:8443           10.8.8.4:41294          ESTABLISHED
tcp        0      0 10.8.8.2:8443           10.8.8.4:43341          ESTABLISHED
tcp        0      0 10.8.8.2:8443           10.8.8.3:47056          ESTABLISHED
tcp        0      0 10.8.8.2:8443           10.8.8.3:40375          ESTABLISHED
tcp        0      0 10.8.8.2:8443           10.8.8.3:40222          ESTABLISHED
tcp        0      0 10.8.8.2:8443           10.8.8.4:38883          ESTABLISHED
tcp        0      0 10.8.8.2:8443           10.8.8.4:41520          ESTABLISHED
tcp        0      0 10.8.8.2:8443           10.8.8.4:40205          ESTABLISHED
tcp        0      0 10.8.8.2:8443           10.8.8.3:52654          ESTABLISHED
tcp        0      0 10.8.8.2:8443           10.8.8.3:55706          ESTABLISHED
tcp        0      0 10.8.8.2:8443           10.8.8.3:46740          ESTABLISHED
tcp        0      0 10.8.8.2:8443           10.8.8.3:42285          ESTABLISHED
tcp        0      0 10.8.8.2:8443           10.8.8.3:33858          ESTABLISHED
tcp        0      0 10.8.8.2:8443           10.8.8.4:44520          ESTABLISHED
tcp        0      0 10.8.8.2:8443           10.8.8.4:56125          ESTABLISHED
tcp        0      0 10.8.8.2:8443           10.8.8.3:32926          ESTABLISHED
tcp        0      0 10.8.8.2:8443           10.8.8.3:37056          ESTABLISHED
tcp        0      0 10.8.8.2:8443           10.8.8.4:40548          ESTABLISHED
1 Like

Hi,
I have this same problem, and I thinks https://discuss.linuxcontainers.org/t/snap-auto-refresh-kills-cluster/2474 is related to this as well.

I have 2 separate clusters of 4 nodes each, in 2 different locations, and they were both unusable this morning.

On one cluster I solved by killing and restarting lxd (see https://discuss.linuxcontainers.org/t/lots-of-log-process-and-lxd-hangs/1592/2?u=sycolth)
(note that the lxc is not affected)
The commands I executed are:

systemctl kill snap.lxd.daemon
pkill lxd # or any other way to kill all those dangling lxd processes if any is left
systemctl start snap.lxd.daemon
systemctl status snap.lxd.daemon

On the other cluster I’m happy to test a more humane solution, if there’s any.

Unfortunately, I’m suffering this issue as well. My cluster is drowning in rapidly-increasing connections on every host, and it’s entirely unresponsive. I attempted to stop and restart the LXD process on each machine but with no luck. Meanwhile the connections climb into the thousands and I have no idea how to fix it.