LXD dies when deploying a container

Pucky_wins · October 6, 2020, 6:48am

Hi

When I launch a container the target lxd server dies. I’m not sure what’s going on. Any ideas?
I run snap 4.6. Below are the logs.

Oct 06 00:55:04 container4 systemd[1]: snap.lxd.daemon.service: Main process exited, code=exited, status=1/FAILURE
Oct 06 00:55:04 container4 systemd[1]: snap.lxd.daemon.service: Failed with result ‘exit-code’.
Oct 06 00:55:05 container4 systemd[1]: snap.lxd.daemon.service: Service hold-off time over, scheduling restart.
Oct 06 00:55:05 container4 systemd[1]: snap.lxd.daemon.service: Scheduled restart job, restart counter is at 16.
Oct 06 00:55:05 container4 systemd[1]: Stopped Service for snap application lxd.daemon.
Oct 06 00:55:05 container4 systemd[1]: Started Service for snap application lxd.daemon.

tomp · October 6, 2020, 7:52am

Please can you enable debug mode:

sudo snap set lxd daemon.debug=true
sudo systemctl reload snap.lxd.daemon

And then try again and send recent output from /var/snap/lxd/common/lxd/logs/lxd.log.

Also please can you show the full command you are running that causes the crash.

Pucky_wins · October 6, 2020, 9:00am

Here you go. It worked the first time after the reload and has failed since. This happens on 3 out of my four container servers running the same cluster.

lxc launch sandbox1:ubuntu20.04-minimal-autoupdate-prod cluster:test1234 --target=container4

Oct 06 10:50:54 container4 systemd[1]: snap.lxd.daemon.service: Main process exited, code=exited, status=1/FAILURE
Oct 06 10:50:54 container4 systemd[1]: snap.lxd.daemon.service: Failed with result 'exit-code'.
Oct 06 10:50:54 container4 systemd[1]: snap.lxd.daemon.service: Service hold-off time over, scheduling restart.
Oct 06 10:50:54 container4 systemd[1]: snap.lxd.daemon.service: Scheduled restart job, restart counter is at 6.
Oct 06 10:50:54 container4 systemd[1]: Stopped Service for snap application lxd.daemon.
Oct 06 10:50:54 container4 systemd[1]: Started Service for snap application lxd.daemon.
Oct 06 10:50:54 container4 lxd.daemon[5815]: => Preparing the system (17597)
Oct 06 10:50:54 container4 lxd.daemon[5815]: ==> Loading snap configuration
Oct 06 10:50:54 container4 lxd.daemon[5815]: ==> Setting up mntns symlink (mnt:[4026532272])
Oct 06 10:50:54 container4 lxd.daemon[5815]: ==> Setting up kmod wrapper
Oct 06 10:50:54 container4 lxd.daemon[5815]: ==> Preparing /boot
Oct 06 10:50:54 container4 lxd.daemon[5815]: ==> Preparing a clean copy of /run
Oct 06 10:50:54 container4 lxd.daemon[5815]: ==> Preparing /run/bin
Oct 06 10:50:54 container4 lxd.daemon[5815]: ==> Preparing a clean copy of /etc
Oct 06 10:50:54 container4 lxd.daemon[5815]: ==> Preparing a clean copy of /usr/share/misc
Oct 06 10:50:54 container4 lxd.daemon[5815]: ==> Setting up ceph configuration
Oct 06 10:50:54 container4 lxd.daemon[5815]: ==> Setting up LVM configuration
Oct 06 10:50:54 container4 lxd.daemon[5815]: ==> Rotating logs
Oct 06 10:50:54 container4 lxd.daemon[5815]: ==> Setting up ZFS (0.7)
Oct 06 10:50:54 container4 lxd.daemon[5815]: ==> Escaping the systemd cgroups
Oct 06 10:50:54 container4 lxd.daemon[5815]: ====> Detected cgroup V1
Oct 06 10:50:54 container4 lxd.daemon[5815]: ==> Escaping the systemd process resource limits
Oct 06 10:50:54 container4 lxd.daemon[5815]: ==> Enabling CRIU
Oct 06 10:50:54 container4 lxd.daemon[5815]: ==> Disabling shiftfs on this kernel (auto)
Oct 06 10:50:54 container4 lxd.daemon[5815]: => Re-using existing LXCFS
Oct 06 10:50:54 container4 lxd.daemon[5815]: => Starting LXD
Oct 06 10:50:55 container4 lxd.daemon[5815]: t=2020-10-06T10:50:55+0200 lvl=info msg="LXD 4.6 is starting in normal mode" path=/var/snap/lxd/common/lxd
Oct 06 10:50:55 container4 lxd.daemon[5815]: t=2020-10-06T10:50:55+0200 lvl=info msg="Kernel uid/gid map:"
Oct 06 10:50:55 container4 lxd.daemon[5815]: t=2020-10-06T10:50:55+0200 lvl=info msg=" - u 0 0 4294967295"
Oct 06 10:50:55 container4 lxd.daemon[5815]: t=2020-10-06T10:50:55+0200 lvl=info msg=" - g 0 0 4294967295"
Oct 06 10:50:55 container4 lxd.daemon[5815]: t=2020-10-06T10:50:55+0200 lvl=info msg="Configured LXD uid/gid map:"
Oct 06 10:50:55 container4 lxd.daemon[5815]: t=2020-10-06T10:50:55+0200 lvl=info msg=" - u 0 1000000 1000000000"
Oct 06 10:50:55 container4 lxd.daemon[5815]: t=2020-10-06T10:50:55+0200 lvl=info msg=" - g 0 1000000 1000000000"
Oct 06 10:50:55 container4 lxd.daemon[5815]: t=2020-10-06T10:50:55+0200 lvl=info msg="Kernel features:"
Oct 06 10:50:55 container4 lxd.daemon[5815]: t=2020-10-06T10:50:55+0200 lvl=info msg=" - closing multiple file descriptors efficiently: no"
Oct 06 10:50:55 container4 lxd.daemon[5815]: t=2020-10-06T10:50:55+0200 lvl=info msg=" - netnsid-based network retrieval: no"
Oct 06 10:50:55 container4 lxd.daemon[5815]: t=2020-10-06T10:50:55+0200 lvl=info msg=" - pidfds: no"
Oct 06 10:50:55 container4 lxd.daemon[5815]: t=2020-10-06T10:50:55+0200 lvl=info msg=" - uevent injection: no"
Oct 06 10:50:55 container4 lxd.daemon[5815]: t=2020-10-06T10:50:55+0200 lvl=info msg=" - seccomp listener: no"
Oct 06 10:50:55 container4 lxd.daemon[5815]: t=2020-10-06T10:50:55+0200 lvl=info msg=" - seccomp listener continue syscalls: no"
Oct 06 10:50:55 container4 lxd.daemon[5815]: t=2020-10-06T10:50:55+0200 lvl=info msg=" - seccomp listener add file descriptors: no"
Oct 06 10:50:55 container4 lxd.daemon[5815]: t=2020-10-06T10:50:55+0200 lvl=info msg=" - attach to namespaces via pidfds: no"
Oct 06 10:50:55 container4 lxd.daemon[5815]: t=2020-10-06T10:50:55+0200 lvl=info msg=" - safe native terminal allocation : no"
Oct 06 10:50:55 container4 lxd.daemon[5815]: t=2020-10-06T10:50:55+0200 lvl=info msg=" - unprivileged file capabilities: yes"
Oct 06 10:50:55 container4 lxd.daemon[5815]: t=2020-10-06T10:50:55+0200 lvl=info msg=" - cgroup layout: hybrid"
Oct 06 10:50:55 container4 lxd.daemon[5815]: t=2020-10-06T10:50:55+0200 lvl=warn msg=" - Couldn't find the CGroup memory swap accounting, swap limits will be ignored"
Oct 06 10:50:55 container4 lxd.daemon[5815]: t=2020-10-06T10:50:55+0200 lvl=info msg=" - shiftfs support: disabled"
Oct 06 10:50:55 container4 lxd.daemon[5815]: t=2020-10-06T10:50:55+0200 lvl=info msg="Initializing local database"
Oct 06 10:50:55 container4 lxd.daemon[5815]: t=2020-10-06T10:50:55+0200 lvl=dbug msg="Initializing database gateway"
Oct 06 10:50:55 container4 lxd.daemon[5815]: t=2020-10-06T10:50:55+0200 lvl=dbug msg="Start database node" address=10.3.0.41:8443 id=4 role=stand-by

tomp · October 6, 2020, 9:09am

What is the remote you have for cluster in that command above? As the logs you’ve posted there show no indication that container4 got the request at all.

Pucky_wins · October 6, 2020, 9:18am

The remote is called container2. I’ll get the same logs there.

All the logs I posted are from after I launch the container. That happens every time it fails. It must be getting the command.

tomp · October 6, 2020, 9:41am

Can you run a tail -f on /var/snap/lxd/common/lxd/logs/lxd.log before you launch the container, and then try and catch the output before lxd restarts, as those logs aren’t useful I’m afraid.

Pucky_wins · October 6, 2020, 9:48am

Ok, that was just out of journalctl. Here’s the requested.

lxc launch sandbox1:ubuntu20.04-minimal-autoupdate-prod cluster:test12345 --target=container4

CONTAINER2

t=2020-10-06T11:43:45+0200 lvl=dbug msg="Found cert" name=0
t=2020-10-06T11:43:46+0200 lvl=dbug msg="Found cert" name=0
t=2020-10-06T11:43:46+0200 lvl=dbug msg="Found cert" name=0
t=2020-10-06T11:43:46+0200 lvl=dbug msg="Replace current raft nodes with [{ID:1 Address:10.3.0.47:8443 Role:voter} {ID:2 Address:10.3.0.52:8443 Role:voter} {ID:3 Address:10.3.0.48:8443 Role:voter} {ID:4 Address:10.3.0.41:8443 Role:stand-by}]" 
t=2020-10-06T11:43:46+0200 lvl=dbug msg="Partial node list heartbeat received, skipping full update" 
t=2020-10-06T11:43:46+0200 lvl=dbug msg="Found cert" name=0
t=2020-10-06T11:43:46+0200 lvl=dbug msg="Replace current raft nodes with [{ID:1 Address:10.3.0.47:8443 Role:voter} {ID:2 Address:10.3.0.52:8443 Role:voter} {ID:3 Address:10.3.0.48:8443 Role:voter} {ID:4 Address:10.3.0.41:8443 Role:stand-by}]" 
t=2020-10-06T11:43:47+0200 lvl=dbug msg="Found cert" name=0
t=2020-10-06T11:43:47+0200 lvl=dbug msg="Replace current raft nodes with [{ID:4 Address:10.3.0.41:8443 Role:stand-by} {ID:1 Address:10.3.0.47:8443 Role:voter} {ID:2 Address:10.3.0.52:8443 Role:voter} {ID:3 Address:10.3.0.48:8443 Role:voter}]" 
t=2020-10-06T11:43:47+0200 lvl=dbug msg="Found cert" name=0
t=2020-10-06T11:43:47+0200 lvl=dbug msg="Replace current raft nodes with [{ID:4 Address:10.3.0.41:8443 Role:stand-by} {ID:1 Address:10.3.0.47:8443 Role:voter} {ID:2 Address:10.3.0.52:8443 Role:voter} {ID:3 Address:10.3.0.48:8443 Role:voter}]" 
t=2020-10-06T11:43:48+0200 lvl=dbug msg="Partial node list heartbeat received, skipping full update" 
t=2020-10-06T11:43:48+0200 lvl=dbug msg="Found cert" name=0
t=2020-10-06T11:43:48+0200 lvl=dbug msg="Replace current raft nodes with [{ID:1 Address:10.3.0.47:8443 Role:voter} {ID:2 Address:10.3.0.52:8443 Role:voter} {ID:3 Address:10.3.0.48:8443 Role:voter} {ID:4 Address:10.3.0.41:8443 Role:stand-by}]" 
t=2020-10-06T11:43:49+0200 lvl=dbug msg="Found cert" name=0
t=2020-10-06T11:43:49+0200 lvl=dbug msg="Replace current raft nodes with [{ID:3 Address:10.3.0.48:8443 Role:voter} {ID:4 Address:10.3.0.41:8443 Role:stand-by} {ID:1 Address:10.3.0.47:8443 Role:voter} {ID:2 Address:10.3.0.52:8443 Role:voter}]" 
t=2020-10-06T11:43:49+0200 lvl=dbug msg="Partial node list heartbeat received, skipping full update" 
t=2020-10-06T11:43:49+0200 lvl=dbug msg="Found cert" name=ca8952b153649920e8392f80045a34e0306ddb1ac591c5512503e077e80ec12b
t=2020-10-06T11:43:49+0200 lvl=dbug msg=Handling ip=10.3.0.214:60984 method=GET url=/1.0 user=ca8952b153649920e8392f80045a34e0306ddb1ac591c5512503e077e80ec12b
t=2020-10-06T11:43:49+0200 lvl=dbug msg="Found cert" name=ca8952b153649920e8392f80045a34e0306ddb1ac591c5512503e077e80ec12b
t=2020-10-06T11:43:50+0200 lvl=dbug msg="Found cert" name=0
t=2020-10-06T11:43:50+0200 lvl=dbug msg="Replace current raft nodes with [{ID:4 Address:10.3.0.41:8443 Role:stand-by} {ID:1 Address:10.3.0.47:8443 Role:voter} {ID:2 Address:10.3.0.52:8443 Role:voter} {ID:3 Address:10.3.0.48:8443 Role:voter}]" 
t=2020-10-06T11:43:50+0200 lvl=dbug msg="Found cert" name=ca8952b153649920e8392f80045a34e0306ddb1ac591c5512503e077e80ec12b
t=2020-10-06T11:43:50+0200 lvl=dbug msg=Handling ip=10.3.0.214:61120 method=GET url="/1.0/events?project=" user=ca8952b153649920e8392f80045a34e0306ddb1ac591c5512503e077e80ec12b
t=2020-10-06T11:43:50+0200 lvl=dbug msg="New event listener: 621ebc6b-5525-4d08-9691-732961cd0257" 
t=2020-10-06T11:43:50+0200 lvl=dbug msg="Found cert" name=ca8952b153649920e8392f80045a34e0306ddb1ac591c5512503e077e80ec12b
t=2020-10-06T11:43:50+0200 lvl=dbug msg=Handling ip=10.3.0.214:61130 method=POST url=/1.0/containers user=ca8952b153649920e8392f80045a34e0306ddb1ac591c5512503e077e80ec12b
t=2020-10-06T11:43:50+0200 lvl=dbug msg="\n\t{\n\t\t\"architecture\": \"\",\n\t\t\"config\": {},\n\t\t\"devices\": {},\n\t\t\"ephemeral\": false,\n\t\t\"profiles\": null,\n\t\t\"stateful\": false,\n\t\t\"description\": \"\",\n\t\t\"name\": \"test123\",\n\t\t\"source\": {\n\t\t\t\"type\": \"image\",\n\t\t\t\"certificate\": \"-----BEGIN CERTIFICATE-----\\nMIIFTDCCAzSgAwIBAgIQTW6I9i8CkkhQl6NelHV/qTANBgkqhkiG9w0BAQsFADA2\\nMRwwGgYDVQQKExNsaW51eGNvbnRhaW5lcnMub3JnMRYwFAYDVQQDDA1yb290QHNh\\nbmRib3gxMB4XDTE5MTAyMzExMDkwOFoXDTI5MTAyMDExMDkwOFowNjEcMBoGA1UE\\nChMTbGludXhjb250YWluZXJzLm9yZzEWMBQGA1UEAwwNcm9vdEBzYW5kYm94MTCC\\nAiIwDQYJKoZIhvcNAQEBBQADggIPADCCAgoCggIBAMT83tN9KrDyDLeTIApDdr4H\\nnh20w++zbO9G05oEcvVnJGDqiRirrHlNhgQdv1yurZTzRRjiwNYIbLbg67hINzaq\\n1JE7C+9x9yBzhumN1B4woLtM25Iikp+DuV3nnEViBpbCejU0t6yBvB3gP8ShMfrU\\nxnEylXw8i6HNEuYnwklwatv82bgjbqVZHkdkW1rpK8xK7qvAux9z4Y+Vyd2xCBCx\\nZTrf3Jq9XJAvP7te/af2BprTUJrspEHfJfSYSU2luczM4pH39fEWTKZ4c1CMLRHZ\\nwtiWfDt/Y5StLpmQ/9LrVYzLKqkmrYrDucunuNoSwtyLMrpfOnfSsB1TdXSl9zYu\\nAAbGTWIiRqIOGibnjwEDxQ2fLcgzIVvZq9+ZIVuzK5HvdCvlB1qfF4gW8pR0OJtS\\nMCnZA9+c2gbmLEH7Yr0UFofEfuvGLzuCTgBUQHfePP06pCa+IHSdd9FVGPC2GJ7E\\n6+hEevBLZHyo0wGtzKqec0onS0fntpNe1VhsW4f8A2YuaKkj5yekP6Bu+YnkjjJz\\njIQ2njrgJRjzdOdk0tgKlyqArL5r4g2TlRoD/uQ1SqQ0tzJxuzBrPnJk+ntNVHob\\nZlMuJ4aUp4NMQOjvwRHUzzT+piMXtwmcWSkJn0AeBHg/A4oRz1sTjXI3spASXZre\\n8J6wUT1ZuNXblsQi84sjAgMBAAGjVjBUMA4GA1UdDwEB/wQEAwIFoDATBgNVHSUE\\nDDAKBggrBgEFBQcDATAMBgNVHRMBAf8EAjAAMB8GA1UdEQQYMBaCCHNhbmRib3gx\\nhwQKAwAxhwQKAAMBMA0GCSqGSIb3DQEBCwUAA4ICAQB8lz+ZQJTLX4m8n93KffD4\\nc5joI2wFXzd5wms8+FSaDxW5t5aElr6pCUQJdNXPiLRyx8M93is/WTjf37BPbM4h\\nsTzCGT/T8q4Lc5h5hwL3jRO8B/5Oxw/zZT+9ccOO58sGrJRiGbA6b7ErvdWwN7WI\\nHNxTg8hnI+6A1SfLfyN5uiD7mT2VMtoqoHVVN/7MUFpfmvh9pHjlb2rMQkWUZDob\\n2DWLwDPfo2kZLQD21B+UFFB5Ifh0YiBTR5lccsU8/C8aX13Oiht2Q+e60FAulxha\\nBq8cuyGjlZKbk+c+pxvtkYhLvPMWc0ahZtvCyd/XYv09xn4WymOl1FIiG+CCPmsS\\nDJvjpEKOwIiIoG4r4tI+b/WosBa/kHmsFnZ6YxIVzXAFA10EI9HZVoMLPkUjDF5w\\nMaE0IzKWPoeO6asDPtp08pfkW3gtAXwi/snOdW7iv4IN60/ChowY89HMytRAZsWQ\\ni5n/J8UDhBRihu4cGAUKySrZW/hPsik+F/RS6oDrsFEaFZrzibv8lFWavxQPBH6k\\ntQnzR/qEsKJPmtyM0lPST56YbG08sGr7LADCw2p+W9fBrCmTgcfY1TbX5dpxOrl+\\nPf/jJCs7G+/W6AvTqckfgLs+y7UHQAK2+UFyshzjRu3OzjVPUm/fi8mp5tbC/Nno\\nqbIAjruxhdvvWBoyXf1YKw==\\n-----END CERTIFICATE-----\\n\",\n\t\t\t\"fingerprint\": \"1beb2c7db4dc39af999729e141472941f5dc4d6156cfbd50274370912938c83f\",\n\t\t\t\"server\": \"https://sandbox1.connect-mobile.co.za:8443\",\n\t\t\t\"secret\": \"d1806a5ebf57be5d8577a07350502676bb0d72cfd4501ea01eeb0d7f1ea0e03a\",\n\t\t\t\"protocol\": \"lxd\",\n\t\t\t\"mode\": \"pull\"\n\t\t},\n\t\t\"instance_type\": \"\"\n\t}" 
t=2020-10-06T11:43:50+0200 lvl=dbug msg="Responding to instance create" 
t=2020-10-06T11:43:50+0200 lvl=dbug msg="Connecting to a remote LXD over HTTPs" 
t=2020-10-06T11:43:50+0200 lvl=dbug msg="Forward instance post request to 10.3.0.41:8443" 
t=2020-10-06T11:43:50+0200 lvl=dbug msg="Connected to the websocket: wss://10.3.0.41:8443/1.0/events?project=default&target=container4" 
t=2020-10-06T11:43:50+0200 lvl=dbug msg="Sending request to LXD" etag= method=POST url="https://10.3.0.41:8443/1.0/instances?project=default&target=container4"
t=2020-10-06T11:43:50+0200 lvl=dbug msg="\n\t{\n\t\t\"architecture\": \"\",\n\t\t\"config\": {},\n\t\t\"devices\": {},\n\t\t\"ephemeral\": false,\n\t\t\"profiles\": null,\n\t\t\"stateful\": false,\n\t\t\"description\": \"\",\n\t\t\"name\": \"test123\",\n\t\t\"source\": {\n\t\t\t\"type\": \"image\",\n\t\t\t\"certificate\": \"-----BEGIN CERTIFICATE-----\\nMIIFTDCCAzSgAwIBAgIQTW6I9i8CkkhQl6NelHV/qTANBgkqhkiG9w0BAQsFADA2\\nMRwwGgYDVQQKExNsaW51eGNvbnRhaW5lcnMub3JnMRYwFAYDVQQDDA1yb290QHNh\\nbmRib3gxMB4XDTE5MTAyMzExMDkwOFoXDTI5MTAyMDExMDkwOFowNjEcMBoGA1UE\\nChMTbGludXhjb250YWluZXJzLm9yZzEWMBQGA1UEAwwNcm9vdEBzYW5kYm94MTCC\\nAiIwDQYJKoZIhvcNAQEBBQADggIPADCCAgoCggIBAMT83tN9KrDyDLeTIApDdr4H\\nnh20w++zbO9G05oEcvVnJGDqiRirrHlNhgQdv1yurZTzRRjiwNYIbLbg67hINzaq\\n1JE7C+9x9yBzhumN1B4woLtM25Iikp+DuV3nnEViBpbCejU0t6yBvB3gP8ShMfrU\\nxnEylXw8i6HNEuYnwklwatv82bgjbqVZHkdkW1rpK8xK7qvAux9z4Y+Vyd2xCBCx\\nZTrf3Jq9XJAvP7te/af2BprTUJrspEHfJfSYSU2luczM4pH39fEWTKZ4c1CMLRHZ\\nwtiWfDt/Y5StLpmQ/9LrVYzLKqkmrYrDucunuNoSwtyLMrpfOnfSsB1TdXSl9zYu\\nAAbGTWIiRqIOGibnjwEDxQ2fLcgzIVvZq9+ZIVuzK5HvdCvlB1qfF4gW8pR0OJtS\\nMCnZA9+c2gbmLEH7Yr0UFofEfuvGLzuCTgBUQHfePP06pCa+IHSdd9FVGPC2GJ7E\\n6+hEevBLZHyo0wGtzKqec0onS0fntpNe1VhsW4f8A2YuaKkj5yekP6Bu+YnkjjJz\\njIQ2njrgJRjzdOdk0tgKlyqArL5r4g2TlRoD/uQ1SqQ0tzJxuzBrPnJk+ntNVHob\\nZlMuJ4aUp4NMQOjvwRHUzzT+piMXtwmcWSkJn0AeBHg/A4oRz1sTjXI3spASXZre\\n8J6wUT1ZuNXblsQi84sjAgMBAAGjVjBUMA4GA1UdDwEB/wQEAwIFoDATBgNVHSUE\\nDDAKBggrBgEFBQcDATAMBgNVHRMBAf8EAjAAMB8GA1UdEQQYMBaCCHNhbmRib3gx\\nhwQKAwAxhwQKAAMBMA0GCSqGSIb3DQEBCwUAA4ICAQB8lz+ZQJTLX4m8n93KffD4\\nc5joI2wFXzd5wms8+FSaDxW5t5aElr6pCUQJdNXPiLRyx8M93is/WTjf37BPbM4h\\nsTzCGT/T8q4Lc5h5hwL3jRO8B/5Oxw/zZT+9ccOO58sGrJRiGbA6b7ErvdWwN7WI\\nHNxTg8hnI+6A1SfLfyN5uiD7mT2VMtoqoHVVN/7MUFpfmvh9pHjlb2rMQkWUZDob\\n2DWLwDPfo2kZLQD21B+UFFB5Ifh0YiBTR5lccsU8/C8aX13Oiht2Q+e60FAulxha\\nBq8cuyGjlZKbk+c+pxvtkYhLvPMWc0ahZtvCyd/XYv09xn4WymOl1FIiG+CCPmsS\\nDJvjpEKOwIiIoG4r4tI+b/WosBa/kHmsFnZ6YxIVzXAFA10EI9HZVoMLPkUjDF5w\\nMaE0IzKWPoeO6asDPtp08pfkW3gtAXwi/snOdW7iv4IN60/ChowY89HMytRAZsWQ\\ni5n/J8UDhBRihu4cGAUKySrZW/hPsik+F/RS6oDrsFEaFZrzibv8lFWavxQPBH6k\\ntQnzR/qEsKJPmtyM0lPST56YbG08sGr7LADCw2p+W9fBrCmTgcfY1TbX5dpxOrl+\\nPf/jJCs7G+/W6AvTqckfgLs+y7UHQAK2+UFyshzjRu3OzjVPUm/fi8mp5tbC/Nno\\nqbIAjruxhdvvWBoyXf1YKw==\\n-----END CERTIFICATE-----\\n\",\n\t\t\t\"fingerprint\": \"1beb2c7db4dc39af999729e141472941f5dc4d6156cfbd50274370912938c83f\",\n\t\t\t\"server\": \"https://sandbox1.connect-mobile.co.za:8443\",\n\t\t\t\"secret\": \"d1806a5ebf57be5d8577a07350502676bb0d72cfd4501ea01eeb0d7f1ea0e03a\",\n\t\t\t\"protocol\": \"lxd\",\n\t\t\t\"mode\": \"pull\"\n\t\t},\n\t\t\"instance_type\": \"\",\n\t\t\"type\": \"container\"\n\t}" 
t=2020-10-06T11:43:51+0200 lvl=dbug msg="Got operation from LXD" 
t=2020-10-06T11:43:51+0200 lvl=dbug msg="\n\t{\n\t\t\"id\": \"c561bad0-b20a-491f-9e5f-347034742d52\",\n\t\t\"class\": \"task\",\n\t\t\"description\": \"Creating container\",\n\t\t\"created_at\": \"2020-10-06T11:43:50.682270211+02:00\",\n\t\t\"updated_at\": \"2020-10-06T11:43:50.682270211+02:00\",\n\t\t\"status\": \"Running\",\n\t\t\"status_code\": 103,\n\t\t\"resources\": {\n\t\t\t\"containers\": [\n\t\t\t\t\"/1.0/containers/test123\"\n\t\t\t],\n\t\t\t\"instances\": [\n\t\t\t\t\"/1.0/instances/test123\"\n\t\t\t]\n\t\t},\n\t\t\"metadata\": null,\n\t\t\"may_cancel\": false,\n\t\t\"err\": \"\",\n\t\t\"location\": \"container4\"\n\t}" 
t=2020-10-06T11:43:51+0200 lvl=dbug msg="\n\t{\n\t\t\"type\": \"async\",\n\t\t\"status\": \"Operation created\",\n\t\t\"status_code\": 100,\n\t\t\"operation\": \"/1.0/operations/c561bad0-b20a-491f-9e5f-347034742d52?project=default\",\n\t\t\"error_code\": 0,\n\t\t\"error\": \"\",\n\t\t\"metadata\": {\n\t\t\t\"id\": \"c561bad0-b20a-491f-9e5f-347034742d52\",\n\t\t\t\"class\": \"task\",\n\t\t\t\"description\": \"Creating container\",\n\t\t\t\"created_at\": \"2020-10-06T11:43:50.682270211+02:00\",\n\t\t\t\"updated_at\": \"2020-10-06T11:43:50.682270211+02:00\",\n\t\t\t\"status\": \"Running\",\n\t\t\t\"status_code\": 103,\n\t\t\t\"resources\": {\n\t\t\t\t\"containers\": [\n\t\t\t\t\t\"/1.0/containers/test123\"\n\t\t\t\t],\n\t\t\t\t\"instances\": [\n\t\t\t\t\t\"/1.0/instances/test123\"\n\t\t\t\t]\n\t\t\t},\n\t\t\t\"metadata\": null,\n\t\t\t\"may_cancel\": false,\n\t\t\t\"err\": \"\",\n\t\t\t\"location\": \"container4\"\n\t\t}\n\t}" 
t=2020-10-06T11:43:51+0200 lvl=dbug msg="Found cert" name=ca8952b153649920e8392f80045a34e0306ddb1ac591c5512503e077e80ec12b
t=2020-10-06T11:43:51+0200 lvl=dbug msg=Handling ip=10.3.0.214:61134 method=GET url=/1.0/operations/c561bad0-b20a-491f-9e5f-347034742d52 user=ca8952b153649920e8392f80045a34e0306ddb1ac591c5512503e077e80ec12b
t=2020-10-06T11:43:51+0200 lvl=dbug msg="Connecting to a remote LXD over HTTPs" 
t=2020-10-06T11:43:51+0200 lvl=dbug msg="Event listener finished: 256e4ac8-6a7e-46b6-b69c-7a25f43e9882" 
t=2020-10-06T11:43:51+0200 lvl=dbug msg="Disconnected event listener: 256e4ac8-6a7e-46b6-b69c-7a25f43e9882" 
t=2020-10-06T11:43:51+0200 lvl=dbug msg="Found cert" name=ca8952b153649920e8392f80045a34e0306ddb1ac591c5512503e077e80ec12b
t=2020-10-06T11:43:51+0200 lvl=dbug msg=Handling ip=10.3.0.214:61150 method=GET url=/1.0/containers/test123 user=ca8952b153649920e8392f80045a34e0306ddb1ac591c5512503e077e80ec12b
t=2020-10-06T11:43:51+0200 lvl=dbug msg="Found cert" name=ca8952b153649920e8392f80045a34e0306ddb1ac591c5512503e077e80ec12b
t=2020-10-06T11:43:51+0200 lvl=dbug msg=Handling ip=10.3.0.214:61156 method=PUT url=/1.0/containers/test123/state user=ca8952b153649920e8392f80045a34e0306ddb1ac591c5512503e077e80ec12b
t=2020-10-06T11:43:51+0200 lvl=dbug msg="\n\t{\n\t\t\"action\": \"start\",\n\t\t\"timeout\": -1,\n\t\t\"force\": false,\n\t\t\"stateful\": false\n\t}" 
t=2020-10-06T11:43:51+0200 lvl=dbug msg="New task Operation: d75e0b2d-1616-4af4-9b06-a5a841079908" 
t=2020-10-06T11:43:51+0200 lvl=dbug msg="Connecting to a remote LXD over HTTPs" 
t=2020-10-06T11:43:51+0200 lvl=warn msg="Failed to get events from node 10.3.0.41:8443: Unable to connect to: 10.3.0.41:8443" 
t=2020-10-06T11:43:51+0200 lvl=dbug msg="Started task operation: d75e0b2d-1616-4af4-9b06-a5a841079908" 
t=2020-10-06T11:43:51+0200 lvl=dbug msg="Failure for task operation: d75e0b2d-1616-4af4-9b06-a5a841079908: Common start logic: The container is already running" 
t=2020-10-06T11:43:51+0200 lvl=dbug msg="\n\t{\n\t\t\"type\": \"async\",\n\t\t\"status\": \"Operation created\",\n\t\t\"status_code\": 100,\n\t\t\"operation\": \"/1.0/operations/d75e0b2d-1616-4af4-9b06-a5a841079908\",\n\t\t\"error_code\": 0,\n\t\t\"error\": \"\",\n\t\t\"metadata\": {\n\t\t\t\"id\": \"d75e0b2d-1616-4af4-9b06-a5a841079908\",\n\t\t\t\"class\": \"task\",\n\t\t\t\"description\": \"Starting container\",\n\t\t\t\"created_at\": \"2020-10-06T11:43:51.273434988+02:00\",\n\t\t\t\"updated_at\": \"2020-10-06T11:43:51.273434988+02:00\",\n\t\t\t\"status\": \"Failure\",\n\t\t\t\"status_code\": 400,\n\t\t\t\"resources\": {\n\t\t\t\t\"containers\": [\n\t\t\t\t\t\"/1.0/containers/test123\"\n\t\t\t\t]\n\t\t\t},\n\t\t\t\"metadata\": null,\n\t\t\t\"may_cancel\": false,\n\t\t\t\"err\": \"Common start logic: The container is already running\",\n\t\t\t\"location\": \"container2\"\n\t\t}\n\t}" 
t=2020-10-06T11:43:51+0200 lvl=dbug msg="Event listener finished: 621ebc6b-5525-4d08-9691-732961cd0257" 
t=2020-10-06T11:43:51+0200 lvl=dbug msg="Disconnected event listener: 621ebc6b-5525-4d08-9691-732961cd0257" 
t=2020-10-06T11:43:51+0200 lvl=dbug msg="Found cert" name=0
t=2020-10-06T11:43:52+0200 lvl=dbug msg="Connecting to a remote LXD over HTTPs" 
t=2020-10-06T11:43:52+0200 lvl=warn msg="Failed to get events from node 10.3.0.41:8443: Unable to connect to: 10.3.0.41:8443" 
t=2020-10-06T11:43:52+0200 lvl=dbug msg="Found cert" name=0
t=2020-10-06T11:43:53+0200 lvl=dbug msg="Found cert" name=0
t=2020-10-06T11:43:53+0200 lvl=dbug msg="Found cert" name=0
t=2020-10-06T11:43:53+0200 lvl=dbug msg="Found cert" name=0
t=2020-10-06T11:43:53+0200 lvl=dbug msg=Handling ip=10.3.0.41:40720 method=GET url="/1.0/events?project=%2A" user=
t=2020-10-06T11:43:53+0200 lvl=dbug msg="New event listener: 48bc826c-4572-44ab-805b-cf57bb1b6105" 
t=2020-10-06T11:43:53+0200 lvl=dbug msg="Connecting to a remote LXD over HTTPs" 
t=2020-10-06T11:43:53+0200 lvl=dbug msg="Connected to the websocket: wss://10.3.0.41:8443/1.0/events?project=%2A" 
t=2020-10-06T11:43:53+0200 lvl=dbug msg="Listening for events on node 10.3.0.41:8443"

CONTAINER4

t=2020-10-06T11:43:46+0200 lvl=dbug msg="Replace current raft nodes with [{ID:3 Address:10.3.0.48:8443 Role:voter} {ID:4 Address:10.3.0.41:8443 Role:stand-by} {ID:1 Address:10.3.0.47:8443 Role:voter} {ID:2 Address:10.3.0.52:8443 Role:voter}]" 
t=2020-10-06T11:43:46+0200 lvl=dbug msg="Partial node list heartbeat received, skipping full update" 
t=2020-10-06T11:43:46+0200 lvl=dbug msg="Found cert" name=0
t=2020-10-06T11:43:46+0200 lvl=dbug msg="Replace current raft nodes with [{ID:1 Address:10.3.0.47:8443 Role:voter} {ID:2 Address:10.3.0.52:8443 Role:voter} {ID:3 Address:10.3.0.48:8443 Role:voter} {ID:4 Address:10.3.0.41:8443 Role:stand-by}]" 
t=2020-10-06T11:43:48+0200 lvl=dbug msg="Found cert" name=0
t=2020-10-06T11:43:48+0200 lvl=dbug msg="Replace current raft nodes with [{ID:1 Address:10.3.0.47:8443 Role:voter} {ID:2 Address:10.3.0.52:8443 Role:voter} {ID:3 Address:10.3.0.48:8443 Role:voter} {ID:4 Address:10.3.0.41:8443 Role:stand-by}]" 
t=2020-10-06T11:43:48+0200 lvl=dbug msg="Partial node list heartbeat received, skipping full update" 
t=2020-10-06T11:43:48+0200 lvl=dbug msg="Found cert" name=0
t=2020-10-06T11:43:48+0200 lvl=dbug msg="Replace current raft nodes with [{ID:2 Address:10.3.0.52:8443 Role:voter} {ID:3 Address:10.3.0.48:8443 Role:voter} {ID:4 Address:10.3.0.41:8443 Role:stand-by} {ID:1 Address:10.3.0.47:8443 Role:voter}]" 
t=2020-10-06T11:43:49+0200 lvl=dbug msg="Found cert" name=0
t=2020-10-06T11:43:49+0200 lvl=dbug msg="Replace current raft nodes with [{ID:1 Address:10.3.0.47:8443 Role:voter} {ID:2 Address:10.3.0.52:8443 Role:voter} {ID:3 Address:10.3.0.48:8443 Role:voter} {ID:4 Address:10.3.0.41:8443 Role:stand-by}]" 
t=2020-10-06T11:43:49+0200 lvl=dbug msg="Partial node list heartbeat received, skipping full update" 
t=2020-10-06T11:43:50+0200 lvl=dbug msg="Found cert" name=0
t=2020-10-06T11:43:50+0200 lvl=dbug msg="Replace current raft nodes with [{ID:3 Address:10.3.0.48:8443 Role:voter} {ID:4 Address:10.3.0.41:8443 Role:stand-by} {ID:1 Address:10.3.0.47:8443 Role:voter} {ID:2 Address:10.3.0.52:8443 Role:voter}]" 
t=2020-10-06T11:43:50+0200 lvl=dbug msg="Found cert" name=0
t=2020-10-06T11:43:50+0200 lvl=dbug msg=Handling ip=10.3.0.47:39776 method=GET url="/1.0/events?project=default&target=container4" user=
t=2020-10-06T11:43:50+0200 lvl=dbug msg="New event listener: 60e954a5-a6b0-40be-9c41-04933a5f50d0" 
t=2020-10-06T11:43:50+0200 lvl=dbug msg="Found cert" name=0
t=2020-10-06T11:43:50+0200 lvl=dbug msg=Handling ip=10.3.0.47:39778 method=POST url="/1.0/instances?project=default&target=container4" user=
t=2020-10-06T11:43:50+0200 lvl=dbug msg="\n\t{\n\t\t\"architecture\": \"\",\n\t\t\"config\": {},\n\t\t\"devices\": {},\n\t\t\"ephemeral\": false,\n\t\t\"profiles\": null,\n\t\t\"stateful\": false,\n\t\t\"description\": \"\",\n\t\t\"name\": \"test123\",\n\t\t\"source\": {\n\t\t\t\"type\": \"image\",\n\t\t\t\"certificate\": \"-----BEGIN CERTIFICATE-----\\nMIIFTDCCAzSgAwIBAgIQTW6I9i8CkkhQl6NelHV/qTANBgkqhkiG9w0BAQsFADA2\\nMRwwGgYDVQQKExNsaW51eGNvbnRhaW5lcnMub3JnMRYwFAYDVQQDDA1yb290QHNh\\nbmRib3gxMB4XDTE5MTAyMzExMDkwOFoXDTI5MTAyMDExMDkwOFowNjEcMBoGA1UE\\nChMTbGludXhjb250YWluZXJzLm9yZzEWMBQGA1UEAwwNcm9vdEBzYW5kYm94MTCC\\nAiIwDQYJKoZIhvcNAQEBBQADggIPADCCAgoCggIBAMT83tN9KrDyDLeTIApDdr4H\\nnh20w++zbO9G05oEcvVnJGDqiRirrHlNhgQdv1yurZTzRRjiwNYIbLbg67hINzaq\\n1JE7C+9x9yBzhumN1B4woLtM25Iikp+DuV3nnEViBpbCejU0t6yBvB3gP8ShMfrU\\nxnEylXw8i6HNEuYnwklwatv82bgjbqVZHkdkW1rpK8xK7qvAux9z4Y+Vyd2xCBCx\\nZTrf3Jq9XJAvP7te/af2BprTUJrspEHfJfSYSU2luczM4pH39fEWTKZ4c1CMLRHZ\\nwtiWfDt/Y5StLpmQ/9LrVYzLKqkmrYrDucunuNoSwtyLMrpfOnfSsB1TdXSl9zYu\\nAAbGTWIiRqIOGibnjwEDxQ2fLcgzIVvZq9+ZIVuzK5HvdCvlB1qfF4gW8pR0OJtS\\nMCnZA9+c2gbmLEH7Yr0UFofEfuvGLzuCTgBUQHfePP06pCa+IHSdd9FVGPC2GJ7E\\n6+hEevBLZHyo0wGtzKqec0onS0fntpNe1VhsW4f8A2YuaKkj5yekP6Bu+YnkjjJz\\njIQ2njrgJRjzdOdk0tgKlyqArL5r4g2TlRoD/uQ1SqQ0tzJxuzBrPnJk+ntNVHob\\nZlMuJ4aUp4NMQOjvwRHUzzT+piMXtwmcWSkJn0AeBHg/A4oRz1sTjXI3spASXZre\\n8J6wUT1ZuNXblsQi84sjAgMBAAGjVjBUMA4GA1UdDwEB/wQEAwIFoDATBgNVHSUE\\nDDAKBggrBgEFBQcDATAMBgNVHRMBAf8EAjAAMB8GA1UdEQQYMBaCCHNhbmRib3gx\\nhwQKAwAxhwQKAAMBMA0GCSqGSIb3DQEBCwUAA4ICAQB8lz+ZQJTLX4m8n93KffD4\\nc5joI2wFXzd5wms8+FSaDxW5t5aElr6pCUQJdNXPiLRyx8M93is/WTjf37BPbM4h\\nsTzCGT/T8q4Lc5h5hwL3jRO8B/5Oxw/zZT+9ccOO58sGrJRiGbA6b7ErvdWwN7WI\\nHNxTg8hnI+6A1SfLfyN5uiD7mT2VMtoqoHVVN/7MUFpfmvh9pHjlb2rMQkWUZDob\\n2DWLwDPfo2kZLQD21B+UFFB5Ifh0YiBTR5lccsU8/C8aX13Oiht2Q+e60FAulxha\\nBq8cuyGjlZKbk+c+pxvtkYhLvPMWc0ahZtvCyd/XYv09xn4WymOl1FIiG+CCPmsS\\nDJvjpEKOwIiIoG4r4tI+b/WosBa/kHmsFnZ6YxIVzXAFA10EI9HZVoMLPkUjDF5w\\nMaE0IzKWPoeO6asDPtp08pfkW3gtAXwi/snOdW7iv4IN60/ChowY89HMytRAZsWQ\\ni5n/J8UDhBRihu4cGAUKySrZW/hPsik+F/RS6oDrsFEaFZrzibv8lFWavxQPBH6k\\ntQnzR/qEsKJPmtyM0lPST56YbG08sGr7LADCw2p+W9fBrCmTgcfY1TbX5dpxOrl+\\nPf/jJCs7G+/W6AvTqckfgLs+y7UHQAK2+UFyshzjRu3OzjVPUm/fi8mp5tbC/Nno\\nqbIAjruxhdvvWBoyXf1YKw==\\n-----END CERTIFICATE-----\\n\",\n\t\t\t\"fingerprint\": \"1beb2c7db4dc39af999729e141472941f5dc4d6156cfbd50274370912938c83f\",\n\t\t\t\"server\": \"https://sandbox1.connect-mobile.co.za:8443\",\n\t\t\t\"secret\": \"d1806a5ebf57be5d8577a07350502676bb0d72cfd4501ea01eeb0d7f1ea0e03a\",\n\t\t\t\"protocol\": \"lxd\",\n\t\t\t\"mode\": \"pull\"\n\t\t},\n\t\t\"instance_type\": \"\",\n\t\t\"type\": \"container\"\n\t}" 
t=2020-10-06T11:43:50+0200 lvl=dbug msg="Responding to instance create" 
t=2020-10-06T11:43:50+0200 lvl=dbug msg="Found cert" name=0
t=2020-10-06T11:43:50+0200 lvl=dbug msg="Replace current raft nodes with [{ID:3 Address:10.3.0.48:8443 Role:voter} {ID:4 Address:10.3.0.41:8443 Role:stand-by} {ID:1 Address:10.3.0.47:8443 Role:voter} {ID:2 Address:10.3.0.52:8443 Role:voter}]" 
t=2020-10-06T11:43:51+0200 lvl=dbug msg="New task Operation: c561bad0-b20a-491f-9e5f-347034742d52" 
t=2020-10-06T11:43:51+0200 lvl=dbug msg="Started task operation: c561bad0-b20a-491f-9e5f-347034742d52" 
t=2020-10-06T11:43:51+0200 lvl=dbug msg="\n\t{\n\t\t\"type\": \"async\",\n\t\t\"status\": \"Operation created\",\n\t\t\"status_code\": 100,\n\t\t\"operation\": \"/1.0/operations/c561bad0-b20a-491f-9e5f-347034742d52\",\n\t\t\"error_code\": 0,\n\t\t\"error\": \"\",\n\t\t\"metadata\": {\n\t\t\t\"id\": \"c561bad0-b20a-491f-9e5f-347034742d52\",\n\t\t\t\"class\": \"task\",\n\t\t\t\"description\": \"Creating container\",\n\t\t\t\"created_at\": \"2020-10-06T11:43:50.682270211+02:00\",\n\t\t\t\"updated_at\": \"2020-10-06T11:43:50.682270211+02:00\",\n\t\t\t\"status\": \"Running\",\n\t\t\t\"status_code\": 103,\n\t\t\t\"resources\": {\n\t\t\t\t\"containers\": [\n\t\t\t\t\t\"/1.0/containers/test123\"\n\t\t\t\t],\n\t\t\t\t\"instances\": [\n\t\t\t\t\t\"/1.0/instances/test123\"\n\t\t\t\t]\n\t\t\t},\n\t\t\t\"metadata\": null,\n\t\t\t\"may_cancel\": false,\n\t\t\t\"err\": \"\",\n\t\t\t\"location\": \"container4\"\n\t\t}\n\t}" 
t=2020-10-06T11:43:51+0200 lvl=dbug msg="Connecting to a remote public LXD over HTTPs" 
t=2020-10-06T11:43:51+0200 lvl=dbug msg="Sending request to LXD" etag= method=GET url=https://sandbox1.connect-mobile.co.za:8443/1.0
t=2020-10-06T11:43:51+0200 lvl=dbug msg="Got response struct from LXD" 
t=2020-10-06T11:43:51+0200 lvl=dbug msg="\n\t{\n\t\t\"config\": null,\n\t\t\"api_extensions\": [\n\t\t\t\"storage_zfs_remove_snapshots\",\n\t\t\t\"container_host_shutdown_timeout\",\n\t\t\t\"container_stop_priority\",\n\t\t\t\"container_syscall_filtering\",\n\t\t\t\"auth_pki\",\n\t\t\t\"container_last_used_at\",\n\t\t\t\"etag\",\n\t\t\t\"patch\",\n\t\t\t\"usb_devices\",\n\t\t\t\"https_allowed_credentials\",\n\t\t\t\"image_compression_algorithm\",\n\t\t\t\"directory_manipulation\",\n\t\t\t\"container_cpu_time\",\n\t\t\t\"storage_zfs_use_refquota\",\n\t\t\t\"storage_lvm_mount_options\",\n\t\t\t\"network\",\n\t\t\t\"profile_usedby\",\n\t\t\t\"container_push\",\n\t\t\t\"container_exec_recording\",\n\t\t\t\"certificate_update\",\n\t\t\t\"container_exec_signal_handling\",\n\t\t\t\"gpu_devices\",\n\t\t\t\"container_image_properties\",\n\t\t\t\"migration_progress\",\n\t\t\t\"id_map\",\n\t\t\t\"network_firewall_filtering\",\n\t\t\t\"network_routes\",\n\t\t\t\"storage\",\n\t\t\t\"file_delete\",\n\t\t\t\"file_append\",\n\t\t\t\"network_dhcp_expiry\",\n\t\t\t\"storage_lvm_vg_rename\",\n\t\t\t\"storage_lvm_thinpool_rename\",\n\t\t\t\"network_vlan\",\n\t\t\t\"image_create_aliases\",\n\t\t\t\"container_stateless_copy\",\n\t\t\t\"container_only_migration\",\n\t\t\t\"storage_zfs_clone_copy\",\n\t\t\t\"unix_device_rename\",\n\t\t\t\"storage_lvm_use_thinpool\",\n\t\t\t\"storage_rsync_bwlimit\",\n\t\t\t\"network_vxlan_interface\",\n\t\t\t\"storage_btrfs_mount_options\",\n\t\t\t\"entity_description\",\n\t\t\t\"image_force_refresh\",\n\t\t\t\"storage_lvm_lv_resizing\",\n\t\t\t\"id_map_base\",\n\t\t\t\"file_symlinks\",\n\t\t\t\"container_push_target\",\n\t\t\t\"network_vlan_physical\",\n\t\t\t\"storage_images_delete\",\n\t\t\t\"container_edit_metadata\",\n\t\t\t\"container_snapshot_stateful_migration\",\n\t\t\t\"storage_driver_ceph\",\n\t\t\t\"storage_ceph_user_name\",\n\t\t\t\"resource_limits\",\n\t\t\t\"storage_volatile_initial_source\",\n\t\t\t\"storage_ceph_force_osd_reuse\",\n\t\t\t\"storage_block_filesystem_btrfs\",\n\t\t\t\"resources\",\n\t\t\t\"kernel_limits\",\n\t\t\t\"storage_api_volume_rename\",\n\t\t\t\"macaroon_authentication\",\n\t\t\t\"network_sriov\",\n\t\t\t\"console\",\n\t\t\t\"restrict_devlxd\",\n\t\t\t\"migration_pre_copy\",\n\t\t\t\"infiniband\",\n\t\t\t\"maas_network\",\n\t\t\t\"devlxd_events\",\n\t\t\t\"proxy\",\n\t\t\t\"network_dhcp_gateway\",\n\t\t\t\"file_get_symlink\",\n\t\t\t\"network_leases\",\n\t\t\t\"unix_device_hotplug\",\n\t\t\t\"storage_api_local_volume_handling\",\n\t\t\t\"operation_description\",\n\t\t\t\"clustering\",\n\t\t\t\"event_lifecycle\",\n\t\t\t\"storage_api_remote_volume_handling\",\n\t\t\t\"nvidia_runtime\",\n\t\t\t\"container_mount_propagation\",\n\t\t\t\"container_backup\",\n\t\t\t\"devlxd_images\",\n\t\t\t\"container_local_cross_pool_handling\",\n\t\t\t\"proxy_unix\",\n\t\t\t\"proxy_udp\",\n\t\t\t\"clustering_join\",\n\t\t\t\"proxy_tcp_udp_multi_port_handling\",\n\t\t\t\"network_state\",\n\t\t\t\"proxy_unix_dac_properties\",\n\t\t\t\"container_protection_delete\",\n\t\t\t\"unix_priv_drop\",\n\t\t\t\"pprof_http\",\n\t\t\t\"proxy_haproxy_protocol\",\n\t\t\t\"network_hwaddr\",\n\t\t\t\"proxy_nat\",\n\t\t\t\"network_nat_order\",\n\t\t\t\"container_full\",\n\t\t\t\"candid_authentication\",\n\t\t\t\"backup_compression\",\n\t\t\t\"candid_config\",\n\t\t\t\"nvidia_runtime_config\",\n\t\t\t\"storage_api_volume_snapshots\",\n\t\t\t\"storage_unmapped\",\n\t\t\t\"projects\",\n\t\t\t\"candid_config_key\",\n\t\t\t\"network_vxlan_ttl\",\n\t\t\t\"container_incremental_copy\",\n\t\t\t\"usb_optional_vendorid\",\n\t\t\t\"snapshot_scheduling\",\n\t\t\t\"container_copy_project\",\n\t\t\t\"clustering_server_address\",\n\t\t\t\"clustering_image_replication\",\n\t\t\t\"container_protection_shift\",\n\t\t\t\"snapshot_expiry\",\n\t\t\t\"container_backup_override_pool\",\n\t\t\t\"snapshot_expiry_creation\",\n\t\t\t\"network_leases_location\",\n\t\t\t\"resources_cpu_socket\",\n\t\t\t\"resources_gpu\",\n\t\t\t\"resources_numa\",\n\t\t\t\"kernel_features\",\n\t\t\t\"id_map_current\",\n\t\t\t\"event_location\",\n\t\t\t\"storage_api_remote_volume_snapshots\",\n\t\t\t\"network_nat_address\",\n\t\t\t\"container_nic_routes\",\n\t\t\t\"rbac\",\n\t\t\t\"cluster_internal_copy\",\n\t\t\t\"seccomp_notify\",\n\t\t\t\"lxc_features\",\n\t\t\t\"container_nic_ipvlan\",\n\t\t\t\"network_vlan_sriov\",\n\t\t\t\"storage_cephfs\",\n\t\t\t\"container_nic_ipfilter\",\n\t\t\t\"resources_v2\",\n\t\t\t\"container_exec_user_group_cwd\",\n\t\t\t\"container_syscall_intercept\",\n\t\t\t\"container_disk_shift\",\n\t\t\t\"storage_shifted\",\n\t\t\t\"resources_infiniband\",\n\t\t\t\"daemon_storage\",\n\t\t\t\"instances\",\n\t\t\t\"image_types\",\n\t\t\t\"resources_disk_sata\",\n\t\t\t\"clustering_roles\",\n\t\t\t\"images_expiry\",\n\t\t\t\"resources_network_firmware\",\n\t\t\t\"backup_compression_algorithm\",\n\t\t\t\"ceph_data_pool_name\",\n\t\t\t\"container_syscall_intercept_mount\",\n\t\t\t\"compression_squashfs\",\n\t\t\t\"container_raw_mount\",\n\t\t\t\"container_nic_routed\",\n\t\t\t\"container_syscall_intercept_mount_fuse\",\n\t\t\t\"container_disk_ceph\",\n\t\t\t\"virtual-machines\",\n\t\t\t\"image_profiles\",\n\t\t\t\"clustering_architecture\",\n\t\t\t\"resources_disk_id\",\n\t\t\t\"storage_lvm_stripes\",\n\t\t\t\"vm_boot_priority\",\n\t\t\t\"unix_hotplug_devices\",\n\t\t\t\"api_filtering\",\n\t\t\t\"instance_nic_network\",\n\t\t\t\"clustering_sizing\",\n\t\t\t\"firewall_driver\",\n\t\t\t\"projects_limits\",\n\t\t\t\"container_syscall_intercept_hugetlbfs\",\n\t\t\t\"limits_hugepages\",\n\t\t\t\"container_nic_routed_gateway\",\n\t\t\t\"projects_restrictions\",\n\t\t\t\"custom_volume_snapshot_expiry\",\n\t\t\t\"volume_snapshot_scheduling\",\n\t\t\t\"trust_ca_certificates\",\n\t\t\t\"snapshot_disk_usage\",\n\t\t\t\"clustering_edit_roles\",\n\t\t\t\"container_nic_routed_host_address\",\n\t\t\t\"container_nic_ipvlan_gateway\",\n\t\t\t\"resources_usb_pci\",\n\t\t\t\"resources_cpu_threads_numa\",\n\t\t\t\"resources_cpu_core_die\",\n\t\t\t\"api_os\",\n\t\t\t\"container_nic_routed_host_table\",\n\t\t\t\"container_nic_ipvlan_host_table\",\n\t\t\t\"container_nic_ipvlan_mode\",\n\t\t\t\"resources_system\",\n\t\t\t\"images_push_relay\",\n\t\t\t\"network_dns_search\",\n\t\t\t\"container_nic_routed_limits\",\n\t\t\t\"instance_nic_bridged_vlan\",\n\t\t\t\"network_state_bond_bridge\",\n\t\t\t\"usedby_consistency\",\n\t\t\t\"custom_block_volumes\",\n\t\t\t\"clustering_failure_domains\",\n\t\t\t\"resources_gpu_mdev\",\n\t\t\t\"console_vga_type\",\n\t\t\t\"projects_limits_disk\",\n\t\t\t\"network_type_macvlan\",\n\t\t\t\"network_type_sriov\",\n\t\t\t\"container_syscall_intercept_bpf_devices\",\n\t\t\t\"network_type_ovn\",\n\t\t\t\"projects_networks\",\n\t\t\t\"projects_networks_restricted_uplinks\"\n\t\t],\n\t\t\"api_status\": \"stable\",\n\t\t\"api_version\": \"1.0\",\n\t\t\"auth\": \"untrusted\",\n\t\t\"public\": false,\n\t\t\"auth_methods\": [\n\t\t\t\"tls\"\n\t\t],\n\t\t\"environment\": {\n\t\t\t\"addresses\": null,\n\t\t\t\"architectures\": null,\n\t\t\t\"certificate\": \"\",\n\t\t\t\"certificate_fingerprint\": \"\",\n\t\t\t\"driver\": \"\",\n\t\t\t\"driver_version\": \"\",\n\t\t\t\"firewall\": \"\",\n\t\t\t\"kernel\": \"\",\n\t\t\t\"kernel_architecture\": \"\",\n\t\t\t\"kernel_features\": null,\n\t\t\t\"kernel_version\": \"\",\n\t\t\t\"lxc_features\": null,\n\t\t\t\"os_name\": \"\",\n\t\t\t\"os_version\": \"\",\n\t\t\t\"project\": \"\",\n\t\t\t\"server\": \"\",\n\t\t\t\"server_clustered\": false,\n\t\t\t\"server_name\": \"\",\n\t\t\t\"server_pid\": 0,\n\t\t\t\"server_version\": \"\",\n\t\t\t\"storage\": \"\",\n\t\t\t\"storage_version\": \"\"\n\t\t}\n\t}" 
tail: /var/snap/lxd/common/lxd/logs/lxd.log: file truncated
t=2020-10-06T11:43:52+0200 lvl=info msg="LXD 4.6 is starting in normal mode" path=/var/snap/lxd/common/lxd
t=2020-10-06T11:43:52+0200 lvl=info msg="Kernel uid/gid map:" 
t=2020-10-06T11:43:52+0200 lvl=info msg=" - u 0 0 4294967295" 
t=2020-10-06T11:43:52+0200 lvl=info msg=" - g 0 0 4294967295" 
t=2020-10-06T11:43:52+0200 lvl=info msg="Configured LXD uid/gid map:" 
t=2020-10-06T11:43:52+0200 lvl=info msg=" - u 0 1000000 1000000000" 
t=2020-10-06T11:43:52+0200 lvl=info msg=" - g 0 1000000 1000000000" 
t=2020-10-06T11:43:52+0200 lvl=info msg="Kernel features:" 
t=2020-10-06T11:43:52+0200 lvl=info msg=" - closing multiple file descriptors efficiently: no" 
t=2020-10-06T11:43:52+0200 lvl=info msg=" - netnsid-based network retrieval: no" 
t=2020-10-06T11:43:52+0200 lvl=info msg=" - pidfds: no" 
t=2020-10-06T11:43:52+0200 lvl=info msg=" - uevent injection: no" 
t=2020-10-06T11:43:52+0200 lvl=info msg=" - seccomp listener: no" 
t=2020-10-06T11:43:52+0200 lvl=info msg=" - seccomp listener continue syscalls: no" 
t=2020-10-06T11:43:52+0200 lvl=info msg=" - seccomp listener add file descriptors: no" 
t=2020-10-06T11:43:52+0200 lvl=info msg=" - attach to namespaces via pidfds: no" 
t=2020-10-06T11:43:52+0200 lvl=info msg=" - safe native terminal allocation : no" 
t=2020-10-06T11:43:52+0200 lvl=info msg=" - unprivileged file capabilities: yes" 
t=2020-10-06T11:43:52+0200 lvl=info msg=" - cgroup layout: hybrid" 
t=2020-10-06T11:43:52+0200 lvl=warn msg=" - Couldn't find the CGroup memory swap accounting, swap limits will be ignored" 
t=2020-10-06T11:43:52+0200 lvl=info msg=" - shiftfs support: disabled" 
t=2020-10-06T11:43:52+0200 lvl=info msg="Initializing local database" 
t=2020-10-06T11:43:52+0200 lvl=dbug msg="Initializing database gateway" 
t=2020-10-06T11:43:52+0200 lvl=dbug msg="Start database node" address=10.3.0.41:8443 id=4 role=stand-by
t=2020-10-06T11:43:52+0200 lvl=dbug msg="Detected stale unix socket, deleting" 
t=2020-10-06T11:43:52+0200 lvl=info msg="Starting cluster handler:" 
t=2020-10-06T11:43:52+0200 lvl=info msg="Starting /dev/lxd handler:" 
t=2020-10-06T11:43:52+0200 lvl=info msg=" - binding devlxd socket" socket=/var/snap/lxd/common/lxd/devlxd/sock
t=2020-10-06T11:43:52+0200 lvl=info msg="REST API daemon:" 
t=2020-10-06T11:43:52+0200 lvl=info msg=" - binding Unix socket" inherited=true socket=/var/snap/lxd/common/lxd/unix.socket
t=2020-10-06T11:43:52+0200 lvl=info msg=" - binding TCP socket" socket=10.3.0.41:8443
t=2020-10-06T11:43:52+0200 lvl=info msg="Initializing global database" 
t=2020-10-06T11:43:52+0200 lvl=dbug msg="Found cert" name=0
t=2020-10-06T11:43:52+0200 lvl=warn msg="Dqlite: attempt 0: server 10.3.0.41:8443: no known leader" 
t=2020-10-06T11:43:52+0200 lvl=dbug msg="Dqlite: attempt 0: server 10.3.0.47:8443: connect to reported leader 10.3.0.48:8443" 
t=2020-10-06T11:43:52+0200 lvl=dbug msg="Dqlite: attempt 0: server 10.3.0.47:8443: connected" 
t=2020-10-06T11:43:52+0200 lvl=dbug msg="Firewall detected \"nftables\" incompatibility: Kernel version does not meet minimum requirement of 5" 
t=2020-10-06T11:43:52+0200 lvl=info msg="Firewall loaded driver \"xtables\"" 
t=2020-10-06T11:43:52+0200 lvl=dbug msg="Notify node 10.3.0.47:8443 of state changes" 
t=2020-10-06T11:43:52+0200 lvl=dbug msg="Notify node 10.3.0.52:8443 of state changes" 
t=2020-10-06T11:43:52+0200 lvl=dbug msg="Notify node 10.3.0.48:8443 of state changes" 
t=2020-10-06T11:43:52+0200 lvl=dbug msg="Connecting to a remote LXD over HTTPs" 
t=2020-10-06T11:43:52+0200 lvl=dbug msg="Connecting to a remote LXD over HTTPs" 
t=2020-10-06T11:43:52+0200 lvl=dbug msg="Connecting to a remote LXD over HTTPs" 
t=2020-10-06T11:43:53+0200 lvl=info msg="Initializing storage pools" 
t=2020-10-06T11:43:53+0200 lvl=dbug msg="Initializing and checking storage pool \"local\"" 
t=2020-10-06T11:43:53+0200 lvl=dbug msg="Mount started" driver=btrfs pool=local
t=2020-10-06T11:43:53+0200 lvl=dbug msg="Mount finished" driver=btrfs pool=local
t=2020-10-06T11:43:53+0200 lvl=info msg="Initializing daemon storage mounts" 
t=2020-10-06T11:43:53+0200 lvl=info msg="Initializing networks" 
t=2020-10-06T11:43:53+0200 lvl=dbug msg="New task Operation: 05f9be6d-ded8-4c4d-ae28-213423b3e6ba" 
t=2020-10-06T11:43:53+0200 lvl=info msg="Pruning leftover image files" 
t=2020-10-06T11:43:53+0200 lvl=dbug msg="Started task operation: 05f9be6d-ded8-4c4d-ae28-213423b3e6ba" 
t=2020-10-06T11:43:53+0200 lvl=info msg="Done pruning leftover image files" 
t=2020-10-06T11:43:53+0200 lvl=info msg="Loading daemon configuration" 
t=2020-10-06T11:43:53+0200 lvl=dbug msg="Success for task operation: 05f9be6d-ded8-4c4d-ae28-213423b3e6ba" 
t=2020-10-06T11:43:53+0200 lvl=dbug msg="Initialized inotify with file descriptor 22" 
t=2020-10-06T11:43:53+0200 lvl=dbug msg=Handling ip=@ method=GET url=/1.0 user=
t=2020-10-06T11:43:53+0200 lvl=dbug msg="Connecting to a remote LXD over HTTPs" 
t=2020-10-06T11:43:53+0200 lvl=dbug msg="Skipping image synchronization task since we're not leader" 
t=2020-10-06T11:43:53+0200 lvl=dbug msg="Connected to the websocket: wss://10.3.0.47:8443/1.0/events?project=%2A" 
t=2020-10-06T11:43:53+0200 lvl=dbug msg="Listening for events on node 10.3.0.47:8443" 
t=2020-10-06T11:43:53+0200 lvl=dbug msg="Connecting to a remote LXD over HTTPs" 
t=2020-10-06T11:43:53+0200 lvl=dbug msg="Found cert" name=0
t=2020-10-06T11:43:53+0200 lvl=dbug msg="New task Operation: c542ab10-1e79-41da-88c6-dc34efc8f227" 
t=2020-10-06T11:43:53+0200 lvl=info msg="Pruning expired images" 
t=2020-10-06T11:43:53+0200 lvl=dbug msg="Started task operation: c542ab10-1e79-41da-88c6-dc34efc8f227" 
t=2020-10-06T11:43:53+0200 lvl=dbug msg=Handling ip=@ method=GET url=/internal/ready user=
t=2020-10-06T11:43:53+0200 lvl=info msg="Done pruning expired images" 
t=2020-10-06T11:43:53+0200 lvl=dbug msg="Connected to the websocket: wss://10.3.0.52:8443/1.0/events?project=%2A" 
t=2020-10-06T11:43:53+0200 lvl=dbug msg="Listening for events on node 10.3.0.52:8443" 
t=2020-10-06T11:43:53+0200 lvl=dbug msg="Connecting to a remote LXD over HTTPs" 
t=2020-10-06T11:43:53+0200 lvl=dbug msg="New task Operation: 3a3ed5cc-69da-4dd7-b910-3f97e99adbac" 
t=2020-10-06T11:43:53+0200 lvl=dbug msg="Found cert" name=0
t=2020-10-06T11:43:53+0200 lvl=dbug msg=Handling ip=10.3.0.52:42850 method=GET url="/1.0/events?project=%2A" user=
t=2020-10-06T11:43:53+0200 lvl=dbug msg="Success for task operation: c542ab10-1e79-41da-88c6-dc34efc8f227" 
t=2020-10-06T11:43:53+0200 lvl=dbug msg="Found cert" name=0
t=2020-10-06T11:43:53+0200 lvl=dbug msg=Handling ip=10.3.0.47:39864 method=GET url="/1.0/events?project=%2A" user=
t=2020-10-06T11:43:53+0200 lvl=info msg="Pruning expired instance backups" 
t=2020-10-06T11:43:53+0200 lvl=dbug msg="Started task operation: 3a3ed5cc-69da-4dd7-b910-3f97e99adbac" 
t=2020-10-06T11:43:53+0200 lvl=dbug msg="Success for task operation: 3a3ed5cc-69da-4dd7-b910-3f97e99adbac" 
t=2020-10-06T11:43:53+0200 lvl=info msg="Done pruning expired instance backups" 
t=2020-10-06T11:43:53+0200 lvl=dbug msg="New event listener: 08dbb93e-5295-49b7-98c1-2b899fc34020" 
t=2020-10-06T11:43:53+0200 lvl=dbug msg="New event listener: 8b262079-b00f-4751-93c2-93bacc27f94e" 
t=2020-10-06T11:43:53+0200 lvl=dbug msg="Connected to the websocket: wss://10.3.0.48:8443/1.0/events?project=%2A" 
t=2020-10-06T11:43:53+0200 lvl=dbug msg="Listening for events on node 10.3.0.48:8443" 
t=2020-10-06T11:43:53+0200 lvl=dbug msg="New task Operation: a3b4cd92-80ae-4cc4-8980-add4d33a12c0" 
t=2020-10-06T11:43:53+0200 lvl=dbug msg="New task Operation: 528f7968-b3c8-4959-8735-8641a6191eb4" 
t=2020-10-06T11:43:53+0200 lvl=info msg="Updating instance types" 
t=2020-10-06T11:43:53+0200 lvl=dbug msg="Started task operation: a3b4cd92-80ae-4cc4-8980-add4d33a12c0" 
t=2020-10-06T11:43:53+0200 lvl=info msg="Done updating instance types" 
t=2020-10-06T11:43:53+0200 lvl=dbug msg="New task Operation: c8fac80c-54d6-4f2d-8953-f50bce874238" 
t=2020-10-06T11:43:53+0200 lvl=info msg="Updating images" 
t=2020-10-06T11:43:53+0200 lvl=dbug msg="Started task operation: c8fac80c-54d6-4f2d-8953-f50bce874238" 
t=2020-10-06T11:43:53+0200 lvl=info msg="Expiring log files" 
t=2020-10-06T11:43:53+0200 lvl=dbug msg="Started task operation: 528f7968-b3c8-4959-8735-8641a6191eb4" 
t=2020-10-06T11:43:53+0200 lvl=info msg="Done updating images" 
t=2020-10-06T11:43:53+0200 lvl=info msg="Done expiring log files" 
t=2020-10-06T11:43:53+0200 lvl=dbug msg="Success for task operation: 528f7968-b3c8-4959-8735-8641a6191eb4" 
t=2020-10-06T11:43:53+0200 lvl=dbug msg="Success for task operation: c8fac80c-54d6-4f2d-8953-f50bce874238" 
t=2020-10-06T11:43:53+0200 lvl=dbug msg="Found cert" name=0
t=2020-10-06T11:43:53+0200 lvl=dbug msg=Handling ip=10.3.0.48:44726 method=GET url="/1.0/events?project=%2A" user=
t=2020-10-06T11:43:53+0200 lvl=dbug msg="New event listener: 23088711-c06a-403f-86ec-619d00c21a1a" 
t=2020-10-06T11:43:53+0200 lvl=dbug msg=Handling ip=@ method=GET url=/1.0 user=
t=2020-10-06T11:43:53+0200 lvl=dbug msg=Handling ip=@ method=GET url=/internal/ready user=

tomp · October 6, 2020, 10:12am

Does it happen without the --target flag?

Pucky_wins · October 6, 2020, 10:47am

Yes, I just wanted to make sure it happened on the correct server.

tomp · October 6, 2020, 10:48am

@stgraber do you know how to access a crash stack trace for LXD inside a snap?

stgraber · October 6, 2020, 11:14am

journalctl -u snap.lxd.daemon -n 300 should have it

Pucky_wins · October 6, 2020, 11:15am

journalctl output was my first few posts.

stgraber · October 6, 2020, 11:17am

Right, but the crash would have been in the lines immediately preceeding what you pasted.

Pucky_wins · October 6, 2020, 1:01pm

That was it, there was nothing else in the output. It just repeats each time the same thing.
It did a few other things this time.

Oct 06 14:59:52 container4 systemd[1]: snap.lxd.daemon.service: Main process exited, code=exited, status=1/FAILURE
Oct 06 14:59:52 container4 systemd[1]: snap.lxd.daemon.service: Failed with result 'exit-code'.
Oct 06 14:59:52 container4 systemd[1]: snap.lxd.daemon.service: Service hold-off time over, scheduling restart.
Oct 06 14:59:52 container4 systemd[1]: snap.lxd.daemon.service: Scheduled restart job, restart counter is at 12.
Oct 06 14:59:52 container4 systemd[1]: Stopped Service for snap application lxd.daemon.
Oct 06 14:59:52 container4 systemd[1]: Started Service for snap application lxd.daemon.
Oct 06 14:59:52 container4 lxd.daemon[15931]: => Preparing the system (17597)
Oct 06 14:59:52 container4 lxd.daemon[15931]: ==> Loading snap configuration
Oct 06 14:59:52 container4 lxd.daemon[15931]: ==> Setting up mntns symlink (mnt:[4026532272])
Oct 06 14:59:52 container4 lxd.daemon[15931]: ==> Setting up kmod wrapper
Oct 06 14:59:52 container4 lxd.daemon[15931]: ==> Preparing /boot
Oct 06 14:59:52 container4 lxd.daemon[15931]: ==> Preparing a clean copy of /run
Oct 06 14:59:52 container4 lxd.daemon[15931]: ==> Preparing /run/bin
Oct 06 14:59:52 container4 lxd.daemon[15931]: ==> Preparing a clean copy of /etc
Oct 06 14:59:52 container4 lxd.daemon[15931]: ==> Preparing a clean copy of /usr/share/misc
Oct 06 14:59:52 container4 lxd.daemon[15931]: ==> Setting up ceph configuration
Oct 06 14:59:52 container4 lxd.daemon[15931]: ==> Setting up LVM configuration
Oct 06 14:59:52 container4 lxd.daemon[15931]: ==> Rotating logs
Oct 06 14:59:54 container4 lxd.daemon[15931]: ==> Setting up ZFS (0.7)
Oct 06 14:59:54 container4 lxd.daemon[15931]: ==> Escaping the systemd cgroups
Oct 06 14:59:54 container4 lxd.daemon[15931]: ====> Detected cgroup V1
Oct 06 14:59:54 container4 lxd.daemon[15931]: ==> Escaping the systemd process resource limits
Oct 06 14:59:54 container4 lxd.daemon[15931]: ==> Enabling CRIU
Oct 06 14:59:54 container4 lxd.daemon[15931]: ==> Disabling shiftfs on this kernel (auto)
Oct 06 14:59:54 container4 lxd.daemon[15931]: => Re-using existing LXCFS
Oct 06 14:59:54 container4 lxd.daemon[15931]: => Starting LXD
Oct 06 14:59:54 container4 lxd.daemon[15931]: t=2020-10-06T14:59:54+0200 lvl=info msg="LXD 4.6 is starting in normal mode" path=/var/snap/lxd/common/lxd
Oct 06 14:59:54 container4 lxd.daemon[15931]: t=2020-10-06T14:59:54+0200 lvl=info msg="Kernel uid/gid map:"
Oct 06 14:59:54 container4 lxd.daemon[15931]: t=2020-10-06T14:59:54+0200 lvl=info msg=" - u 0 0 4294967295"
Oct 06 14:59:54 container4 lxd.daemon[15931]: t=2020-10-06T14:59:54+0200 lvl=info msg=" - g 0 0 4294967295"
Oct 06 14:59:54 container4 lxd.daemon[15931]: t=2020-10-06T14:59:54+0200 lvl=info msg="Configured LXD uid/gid map:"
Oct 06 14:59:54 container4 lxd.daemon[15931]: t=2020-10-06T14:59:54+0200 lvl=info msg=" - u 0 1000000 1000000000"
Oct 06 14:59:54 container4 lxd.daemon[15931]: t=2020-10-06T14:59:54+0200 lvl=info msg=" - g 0 1000000 1000000000"
Oct 06 14:59:54 container4 lxd.daemon[15931]: t=2020-10-06T14:59:54+0200 lvl=info msg="Kernel features:"
Oct 06 14:59:54 container4 lxd.daemon[15931]: t=2020-10-06T14:59:54+0200 lvl=info msg=" - closing multiple file descriptors efficiently: no"
Oct 06 14:59:54 container4 lxd.daemon[15931]: t=2020-10-06T14:59:54+0200 lvl=info msg=" - netnsid-based network retrieval: no"
Oct 06 14:59:54 container4 lxd.daemon[15931]: t=2020-10-06T14:59:54+0200 lvl=info msg=" - pidfds: no"
Oct 06 14:59:54 container4 lxd.daemon[15931]: t=2020-10-06T14:59:54+0200 lvl=info msg=" - uevent injection: no"
Oct 06 14:59:54 container4 lxd.daemon[15931]: t=2020-10-06T14:59:54+0200 lvl=info msg=" - seccomp listener: no"
Oct 06 14:59:54 container4 lxd.daemon[15931]: t=2020-10-06T14:59:54+0200 lvl=info msg=" - seccomp listener continue syscalls: no"
Oct 06 14:59:54 container4 lxd.daemon[15931]: t=2020-10-06T14:59:54+0200 lvl=info msg=" - seccomp listener add file descriptors: no"
Oct 06 14:59:54 container4 lxd.daemon[15931]: t=2020-10-06T14:59:54+0200 lvl=info msg=" - attach to namespaces via pidfds: no"
Oct 06 14:59:54 container4 lxd.daemon[15931]: t=2020-10-06T14:59:54+0200 lvl=info msg=" - safe native terminal allocation : no"
Oct 06 14:59:54 container4 lxd.daemon[15931]: t=2020-10-06T14:59:54+0200 lvl=info msg=" - unprivileged file capabilities: yes"
Oct 06 14:59:54 container4 lxd.daemon[15931]: t=2020-10-06T14:59:54+0200 lvl=info msg=" - cgroup layout: hybrid"
Oct 06 14:59:54 container4 lxd.daemon[15931]: t=2020-10-06T14:59:54+0200 lvl=warn msg=" - Couldn't find the CGroup memory swap accounting, swap limits will be ignored"
Oct 06 14:59:54 container4 lxd.daemon[15931]: t=2020-10-06T14:59:54+0200 lvl=info msg=" - shiftfs support: disabled"
Oct 06 14:59:54 container4 lxd.daemon[15931]: t=2020-10-06T14:59:54+0200 lvl=info msg="Initializing local database"
Oct 06 14:59:54 container4 lxd.daemon[15931]: t=2020-10-06T14:59:54+0200 lvl=dbug msg="Initializing database gateway"
Oct 06 14:59:54 container4 lxd.daemon[15931]: t=2020-10-06T14:59:54+0200 lvl=dbug msg="Start database node" address=10.3.0.41:8443 id=4 role=stand-by
Oct 06 14:59:55 container4 lxd.daemon[15931]: t=2020-10-06T14:59:55+0200 lvl=dbug msg="Detected stale unix socket, deleting"
Oct 06 14:59:55 container4 lxd.daemon[15931]: t=2020-10-06T14:59:55+0200 lvl=info msg="Starting cluster handler:"
Oct 06 14:59:55 container4 lxd.daemon[15931]: t=2020-10-06T14:59:55+0200 lvl=info msg="Starting /dev/lxd handler:"
Oct 06 14:59:55 container4 lxd.daemon[15931]: t=2020-10-06T14:59:55+0200 lvl=info msg=" - binding devlxd socket" socket=/var/snap/lxd/common/lxd/devlxd/sock
Oct 06 14:59:55 container4 lxd.daemon[15931]: t=2020-10-06T14:59:55+0200 lvl=info msg="REST API daemon:"
Oct 06 14:59:55 container4 lxd.daemon[15931]: t=2020-10-06T14:59:55+0200 lvl=info msg=" - binding Unix socket" inherited=true socket=/var/snap/lxd/common/lxd/unix.socket
Oct 06 14:59:55 container4 lxd.daemon[15931]: t=2020-10-06T14:59:55+0200 lvl=info msg=" - binding TCP socket" socket=10.3.0.41:8443
Oct 06 14:59:55 container4 lxd.daemon[15931]: t=2020-10-06T14:59:55+0200 lvl=info msg="Initializing global database"
Oct 06 14:59:55 container4 lxd.daemon[15931]: t=2020-10-06T14:59:55+0200 lvl=dbug msg="Found cert" name=0
Oct 06 14:59:55 container4 lxd.daemon[15931]: t=2020-10-06T14:59:55+0200 lvl=warn msg="Dqlite: attempt 0: server 10.3.0.41:8443: no known leader"
Oct 06 14:59:55 container4 lxd.daemon[15931]: t=2020-10-06T14:59:55+0200 lvl=dbug msg="Dqlite: attempt 0: server 10.3.0.47:8443: connect to reported leader 10.3.0.48:8443"
Oct 06 14:59:55 container4 lxd.daemon[15931]: t=2020-10-06T14:59:55+0200 lvl=dbug msg="Dqlite: attempt 0: server 10.3.0.47:8443: connected"

stgraber · October 6, 2020, 1:45pm

Hmm, that’s a bit annoying

If it can easily be reproduced, you could do:

systemctl stop snap.lxd.daemon.service snap.lxd.daemon.unix.socket
lxd --debug --group lxd

To manually have LXD run in a terminal, then use another to trigger the crash.

To go back to normal, after ctrl+c in the terminal running LXD, you can start the units back with:

systemctl start snap.lxd.daemon.unix.socket
lxc info

Pucky_wins · October 6, 2020, 1:49pm

It’s very easy to reproduce. I just try and launch a container to it. It works on the odd occasion but fails mostly. I can’t shut down those containers though, it’s unfortunately my production cluster. Would strace or something be useful?

stgraber · October 6, 2020, 2:04pm

You can try to strace the LXD process to see what it does before a crash, but unless it’s crashing in some C code we call from Go, the trace is unlikely to be very useful.

Pucky_wins · October 6, 2020, 2:06pm

Can I shutdown the daemon without affecting the containers? I used to do that pre-snap. Found out the hard way it doesn’t work now.

stgraber · October 6, 2020, 3:02pm

kill $(cat /var/snap/lxd/common/lxd.pid) should have LXD exit cleanly.
You can check that the unit is then inactivate in systemctl status snap.lxd.daemon.

If it is, then run:

systemctl stop snap.lxd.daemon.unix.socket
lxd --debug --group lxd

Pucky_wins · October 6, 2020, 3:04pm

Thanks, I’ll check it out tonight.