Deleted containers on ZFS getting stuck

I’ve noticed a bug with latest and previous LXD versions that instances will get stuck after deletion is requested, the container will stick around in the RUNNING state.

$ lxc delete -f bc-crate-1d0aa532-add0-40b9-a7a8-b44689be5824
Error: Error deleting storage volume: Failed to run: zfs destroy -r master/containers/bc-crate-1d0aa532-add0-40b9-a7a8-b44689be5824: cannot destroy 'master/containers/bc-crate-1d0aa532-add0-40b9-a7a8-b44689be5824': dataset is busy

After running this command the container is stopped, but the problem still persists trying to delete. I’ve had to use the advice at https://github.com/openzfs/zfs/issues/1810 to kill processes, but this has to be done manually and isn’t ideal. Any ideas why this might be happening?

I’m using LXD 4.7/18137 on the snap package. Here is the containers configuration:

architecture: x86_64
config:
  image.architecture: amd64
  image.description: Debian buster amd64 (20200713_05:24)
  image.name: debian-buster-amd64-default-20200713_05:24
  image.os: debian
  image.release: buster
  image.serial: "20200713_05:24"
  image.variant: default
  limits.cpu: "2"
  limits.memory: 1152MB
  security.devlxd: "false"
  volatile.base_image: d689cfcbfa67b88219c7750b53267b0f01a4c497e8f1d3445773e0b13fe84ca1
  volatile.eth0.hwaddr: 00:16:3e:49:6f:87
  volatile.idmap.base: "0"
  volatile.idmap.current: '[{"Isuid":true,"Isgid":false,"Hostid":1000000,"Nsid":0,"Maprange":1000000000},{"Isuid":false,"Isgid":true,"Hostid":1000000,"Nsid":0,"Maprange":1000000000}]'
  volatile.idmap.next: '[{"Isuid":true,"Isgid":false,"Hostid":1000000,"Nsid":0,"Maprange":1000000000},{"Isuid":false,"Isgid":true,"Hostid":1000000,"Nsid":0,"Maprange":1000000000}]'
  volatile.last_state.idmap: '[{"Isuid":true,"Isgid":false,"Hostid":1000000,"Nsid":0,"Maprange":1000000000},{"Isuid":false,"Isgid":true,"Hostid":1000000,"Nsid":0,"Maprange":1000000000}]'
  volatile.last_state.power: STOPPED
devices:
  crated:
    connect: unix:/run/crated.sock
    gid: "1000"
    listen: unix:/tmp/bc-daemon-1d0aa532-add0-40b9-a7a8-b44689be5824-crated.sock
    type: proxy
    uid: "1000"
  eth0:
    ipv4.address: 10.98.0.11
    name: eth0
    network: lxdbr0
    parent: ""
    type: nic
ephemeral: false
profiles:
- default
stateful: false
description: ""

Usually indicating of a mount namespace bug with the snap.

Can you show journalctl -u snap.lxd.daemon -n 3000?

The log with 3000 lines was too large to post, here is the log containing the issue up to the previous full host restart (not snap refresh). It appears the problem happens at Oct 26 10:44:49.

Oct 22 19:54:56 ns3171943 lxd.daemon[1350021]: t=2020-10-22T19:54:56+0100 lvl=warn msg="Rejecting request from untrusted client" ip=127.0.0.1:55104
Oct 22 20:19:54 ns3171943 systemd[1]: Stopping Service for snap application lxd.daemon...
Oct 22 20:19:54 ns3171943 lxd.daemon[48719]: => Stop reason is: host shutdown
Oct 22 20:19:54 ns3171943 lxd.daemon[48719]: => Stopping LXD (with container shutdown)
Oct 22 20:19:55 ns3171943 lxd.daemon[1350021]: t=2020-10-22T20:19:55+0100 lvl=eror msg="Failed to stop device 'crated': Could not read pid file: Unable to parse Process YAML: yaml: unmarshal errors:\n  line 1: cannot unmarshal !!int `50694` into subprocess.Process"
Oct 22 20:19:55 ns3171943 lxd.daemon[1350021]: t=2020-10-22T20:19:55+0100 lvl=eror msg="Failed to stop device 'crated': Could not read pid file: Unable to parse Process YAML: yaml: unmarshal errors:\n  line 1: cannot unmarshal !!int `3525361` into subprocess.Process"
Oct 22 20:19:55 ns3171943 lxd.daemon[1350021]: t=2020-10-22T20:19:55+0100 lvl=eror msg="Failed to stop device 'crated': Could not read pid file: Unable to parse Process YAML: yaml: unmarshal errors:\n  line 1: cannot unmarshal !!int `2508656` into subprocess.Process"
Oct 22 20:19:56 ns3171943 lxd.daemon[1350021]: t=2020-10-22T20:19:56+0100 lvl=eror msg="Failed to stop device 'crated': Could not read pid file: Unable to parse Process YAML: yaml: unmarshal errors:\n  line 1: cannot unmarshal !!int `3105277` into subprocess.Process"
Oct 22 20:19:56 ns3171943 lxd.daemon[1350021]: t=2020-10-22T20:19:56+0100 lvl=eror msg="Failed to stop device 'crated': Could not read pid file: Unable to parse Process YAML: yaml: unmarshal errors:\n  line 1: cannot unmarshal !!int `2980235` into subprocess.Process"
Oct 22 20:19:57 ns3171943 lxd.daemon[1349535]: => LXD exited cleanly
Oct 22 20:19:58 ns3171943 lxd.daemon[48719]: ==> Stopped LXD
Oct 22 20:19:58 ns3171943 lxd.daemon[48719]: => Stopping LXCFS
Oct 22 20:19:58 ns3171943 lxd.daemon[1687]: Running destructor lxcfs_exit
Oct 22 20:19:59 ns3171943 lxd.daemon[48719]: ==> Stopped LXCFS
Oct 22 20:19:59 ns3171943 lxd.daemon[48719]: => Cleaning up PID files
Oct 22 20:19:59 ns3171943 lxd.daemon[48719]: => Cleaning up namespaces
Oct 22 20:19:59 ns3171943 lxd.daemon[48719]: => All done
Oct 22 20:19:59 ns3171943 systemd[1]: snap.lxd.daemon.service: Succeeded.
Oct 22 20:19:59 ns3171943 systemd[1]: Stopped Service for snap application lxd.daemon.
Oct 22 20:19:59 ns3171943 systemd[1]: Started Service for snap application lxd.daemon.
Oct 22 20:19:59 ns3171943 lxd.daemon[50007]: => Preparing the system (17936)
Oct 22 20:19:59 ns3171943 lxd.daemon[50007]: ==> Loading snap configuration
Oct 22 20:19:59 ns3171943 lxd.daemon[50007]: ==> Setting up mntns symlink (mnt:[4026532681])
Oct 22 20:19:59 ns3171943 lxd.daemon[50007]: ==> Setting up kmod wrapper
Oct 22 20:19:59 ns3171943 lxd.daemon[50007]: ==> Preparing /boot
Oct 22 20:19:59 ns3171943 lxd.daemon[50007]: ==> Preparing a clean copy of /run
Oct 22 20:19:59 ns3171943 lxd.daemon[50007]: ==> Preparing /run/bin
Oct 22 20:19:59 ns3171943 lxd.daemon[50007]: ==> Preparing a clean copy of /etc
Oct 22 20:19:59 ns3171943 lxd.daemon[50007]: ==> Preparing a clean copy of /usr/share/misc
Oct 22 20:19:59 ns3171943 lxd.daemon[50007]: ==> Setting up ceph configuration
Oct 22 20:19:59 ns3171943 lxd.daemon[50007]: ==> Setting up LVM configuration
Oct 22 20:19:59 ns3171943 lxd.daemon[50007]: ==> Rotating logs
Oct 22 20:19:59 ns3171943 lxd.daemon[50007]: ==> Setting up ZFS (0.8)
Oct 22 20:19:59 ns3171943 lxd.daemon[50007]: ==> Escaping the systemd cgroups
Oct 22 20:19:59 ns3171943 lxd.daemon[50007]: ====> Detected cgroup V1
Oct 22 20:19:59 ns3171943 lxd.daemon[50007]: ==> Escaping the systemd process resource limits
Oct 22 20:19:59 ns3171943 lxd.daemon[50007]: ==> Disabling shiftfs on this kernel (auto)
Oct 22 20:19:59 ns3171943 lxd.daemon[50007]: ==> Detected kernel with partial AppArmor support
Oct 22 20:19:59 ns3171943 lxd.daemon[50007]: => Starting LXCFS
Oct 22 20:19:59 ns3171943 lxd.daemon[50126]: Running constructor lxcfs_init to reload liblxcfs
Oct 22 20:19:59 ns3171943 lxd.daemon[50126]: mount namespace: 4
Oct 22 20:19:59 ns3171943 lxd.daemon[50126]: hierarchies:
Oct 22 20:19:59 ns3171943 lxd.daemon[50126]:   0: fd:   5:
Oct 22 20:19:59 ns3171943 lxd.daemon[50126]:   1: fd:   6: name=systemd
Oct 22 20:19:59 ns3171943 lxd.daemon[50126]:   2: fd:   7: freezer
Oct 22 20:19:59 ns3171943 lxd.daemon[50126]:   3: fd:   8: cpu,cpuacct
Oct 22 20:19:59 ns3171943 lxd.daemon[50126]:   4: fd:   9: devices
Oct 22 20:19:59 ns3171943 lxd.daemon[50126]:   5: fd:  10: perf_event
Oct 22 20:19:59 ns3171943 lxd.daemon[50126]:   6: fd:  11: cpuset
Oct 22 20:19:59 ns3171943 lxd.daemon[50126]:   7: fd:  12: net_cls,net_prio
Oct 22 20:19:59 ns3171943 lxd.daemon[50126]:   8: fd:  13: blkio
Oct 22 20:19:59 ns3171943 lxd.daemon[50126]:   9: fd:  14: rdma
Oct 22 20:19:59 ns3171943 lxd.daemon[50126]:  10: fd:  15: pids
Oct 22 20:19:59 ns3171943 lxd.daemon[50126]:  11: fd:  16: memory
Oct 22 20:19:59 ns3171943 lxd.daemon[50126]: Kernel supports pidfds
Oct 22 20:19:59 ns3171943 lxd.daemon[50126]: Kernel does not support swap accounting
Oct 22 20:19:59 ns3171943 lxd.daemon[50126]: api_extensions:
Oct 22 20:19:59 ns3171943 lxd.daemon[50126]: - cgroups
Oct 22 20:19:59 ns3171943 lxd.daemon[50126]: - sys_cpu_online
Oct 22 20:19:59 ns3171943 lxd.daemon[50126]: - proc_cpuinfo
Oct 22 20:19:59 ns3171943 lxd.daemon[50126]: - proc_diskstats
Oct 22 20:19:59 ns3171943 lxd.daemon[50126]: - proc_loadavg
Oct 22 20:19:59 ns3171943 lxd.daemon[50126]: - proc_meminfo
Oct 22 20:19:59 ns3171943 lxd.daemon[50126]: - proc_stat
Oct 22 20:19:59 ns3171943 lxd.daemon[50126]: - proc_swaps
Oct 22 20:19:59 ns3171943 lxd.daemon[50126]: - proc_uptime
Oct 22 20:19:59 ns3171943 lxd.daemon[50126]: - shared_pidns
Oct 22 20:19:59 ns3171943 lxd.daemon[50126]: - cpuview_daemon
Oct 22 20:19:59 ns3171943 lxd.daemon[50126]: - loadavg_daemon
Oct 22 20:19:59 ns3171943 lxd.daemon[50126]: - pidfds
Oct 22 20:20:00 ns3171943 lxd.daemon[50007]: => Starting LXD
Oct 22 20:20:00 ns3171943 lxd.daemon[50137]: t=2020-10-22T20:20:00+0100 lvl=warn msg=" - Couldn't find the CGroup blkio.weight, I/O weight limits will be ignored"
Oct 22 20:20:00 ns3171943 lxd.daemon[50137]: t=2020-10-22T20:20:00+0100 lvl=warn msg=" - Couldn't find the CGroup hugetlb controller, hugepage limits will be ignored"
Oct 22 20:20:00 ns3171943 lxd.daemon[50137]: t=2020-10-22T20:20:00+0100 lvl=warn msg=" - Couldn't find the CGroup memory swap accounting, swap limits will be ignored"
Oct 22 20:20:01 ns3171943 lxd.daemon[50137]: t=2020-10-22T20:20:01+0100 lvl=eror msg="Failed to stop device 'crated': Could not read pid file: Unable to parse Process YAML: yaml: unmarshal errors:\n  line 1: cannot unmarshal !!int `50694` into subprocess.Process"
Oct 22 20:20:01 ns3171943 lxd.daemon[50137]: t=2020-10-22T20:20:01+0100 lvl=eror msg="Failed starting container" action=start created=2020-08-26T10:12:26+0100 ephemeral=false name=bc-crate-1ae3b69d-b903-4fd3-9c63-cbf940d7f8df project=default stateful=false used=2020-08-26T10:12:27+0100
Oct 22 20:20:01 ns3171943 lxd.daemon[50137]: t=2020-10-22T20:20:01+0100 lvl=eror msg="Failed to start instance 'bc-crate-1ae3b69d-b903-4fd3-9c63-cbf940d7f8df': Failed to run: /snap/lxd/current/bin/lxd forkstart bc-crate-1ae3b69d-b903-4fd3-9c63-cbf940d7f8df /var/snap/lxd/common/lxd/containers /var/snap/lxd/common/lxd/logs/bc-crate-1ae3b69d-b903-4fd3-9c63-cbf940d7f8df/lxc.conf: "
Oct 22 20:20:01 ns3171943 lxd.daemon[50137]: t=2020-10-22T20:20:01+0100 lvl=eror msg="Failed to stop device 'crated': Could not read pid file: Unable to parse Process YAML: yaml: unmarshal errors:\n  line 1: cannot unmarshal !!int `2508656` into subprocess.Process"
Oct 22 20:20:01 ns3171943 lxd.daemon[50137]: t=2020-10-22T20:20:01+0100 lvl=eror msg="Failed starting container" action=start created=2020-09-02T19:00:57+0100 ephemeral=false name=bc-crate-28b3efb7-072b-45c8-9d28-a0263c6daea9 project=default stateful=false used=2020-09-02T19:00:58+0100
Oct 22 20:20:01 ns3171943 lxd.daemon[50137]: t=2020-10-22T20:20:01+0100 lvl=eror msg="Failed to start instance 'bc-crate-28b3efb7-072b-45c8-9d28-a0263c6daea9': Failed to run: /snap/lxd/current/bin/lxd forkstart bc-crate-28b3efb7-072b-45c8-9d28-a0263c6daea9 /var/snap/lxd/common/lxd/containers /var/snap/lxd/common/lxd/logs/bc-crate-28b3efb7-072b-45c8-9d28-a0263c6daea9/lxc.conf: "
Oct 22 20:20:02 ns3171943 lxd.daemon[50137]: t=2020-10-22T20:20:02+0100 lvl=eror msg="Failed to stop device 'crated': Could not read pid file: Unable to parse Process YAML: yaml: unmarshal errors:\n  line 1: cannot unmarshal !!int `3525361` into subprocess.Process"
Oct 22 20:20:02 ns3171943 lxd.daemon[50137]: t=2020-10-22T20:20:02+0100 lvl=eror msg="Failed starting container" action=start created=2020-09-08T15:47:29+0100 ephemeral=false name=bc-crate-5278224a-a23c-4fd1-a726-1c30694cc80a project=default stateful=false used=2020-09-08T15:47:30+0100
Oct 22 20:20:02 ns3171943 lxd.daemon[50137]: t=2020-10-22T20:20:02+0100 lvl=eror msg="Failed to start instance 'bc-crate-5278224a-a23c-4fd1-a726-1c30694cc80a': Failed to run: /snap/lxd/current/bin/lxd forkstart bc-crate-5278224a-a23c-4fd1-a726-1c30694cc80a /var/snap/lxd/common/lxd/containers /var/snap/lxd/common/lxd/logs/bc-crate-5278224a-a23c-4fd1-a726-1c30694cc80a/lxc.conf: "
Oct 22 20:20:02 ns3171943 lxd.daemon[50137]: t=2020-10-22T20:20:02+0100 lvl=eror msg="Failed to stop device 'crated': Could not read pid file: Unable to parse Process YAML: yaml: unmarshal errors:\n  line 1: cannot unmarshal !!int `2980235` into subprocess.Process"
Oct 22 20:20:02 ns3171943 lxd.daemon[50137]: t=2020-10-22T20:20:02+0100 lvl=eror msg="Failed starting container" action=start created=2020-08-21T13:32:22+0100 ephemeral=false name=bc-crate-868a0ee0-5e9f-4d45-bbb9-088bc7be3cf8 project=default stateful=false used=2020-08-21T13:32:22+0100
Oct 22 20:20:02 ns3171943 lxd.daemon[50137]: t=2020-10-22T20:20:02+0100 lvl=eror msg="Failed to start instance 'bc-crate-868a0ee0-5e9f-4d45-bbb9-088bc7be3cf8': Failed to run: /snap/lxd/current/bin/lxd forkstart bc-crate-868a0ee0-5e9f-4d45-bbb9-088bc7be3cf8 /var/snap/lxd/common/lxd/containers /var/snap/lxd/common/lxd/logs/bc-crate-868a0ee0-5e9f-4d45-bbb9-088bc7be3cf8/lxc.conf: "
Oct 22 20:20:05 ns3171943 lxd.daemon[50137]: t=2020-10-22T20:20:05+0100 lvl=eror msg="Failed to stop device 'crated': Could not read pid file: Unable to parse Process YAML: yaml: unmarshal errors:\n  line 1: cannot unmarshal !!int `3105277` into subprocess.Process"
Oct 22 20:20:05 ns3171943 lxd.daemon[50137]: t=2020-10-22T20:20:05+0100 lvl=eror msg="Failed starting container" action=start created=2020-08-23T20:41:33+0100 ephemeral=false name=bc-crate-fdf5b1b1-7654-461f-a3d8-346830ce437d project=default stateful=false used=2020-08-23T20:41:34+0100
Oct 22 20:20:05 ns3171943 lxd.daemon[50137]: t=2020-10-22T20:20:05+0100 lvl=eror msg="Failed to start instance 'bc-crate-fdf5b1b1-7654-461f-a3d8-346830ce437d': Failed to run: /snap/lxd/current/bin/lxd forkstart bc-crate-fdf5b1b1-7654-461f-a3d8-346830ce437d /var/snap/lxd/common/lxd/containers /var/snap/lxd/common/lxd/logs/bc-crate-fdf5b1b1-7654-461f-a3d8-346830ce437d/lxc.conf: "
Oct 22 20:20:05 ns3171943 lxd.daemon[50007]: => LXD is ready
Oct 22 20:20:31 ns3171943 lxd.daemon[50137]: t=2020-10-22T20:20:31+0100 lvl=eror msg="Failed to stop device 'crated': Could not read pid file: Unable to parse Process YAML: yaml: unmarshal errors:\n  line 1: cannot unmarshal !!int `50694` into subprocess.Process"
Oct 22 20:20:31 ns3171943 lxd.daemon[50137]: t=2020-10-22T20:20:31+0100 lvl=eror msg="Failed starting container" action=start created=2020-08-26T10:12:26+0100 ephemeral=false name=bc-crate-1ae3b69d-b903-4fd3-9c63-cbf940d7f8df project=default stateful=false used=2020-10-22T20:20:01+0100
Oct 22 20:21:35 ns3171943 lxd.daemon[50137]: t=2020-10-22T20:21:35+0100 lvl=eror msg="Failed to stop device 'crated': Could not read pid file: Unable to parse Process YAML: yaml: unmarshal errors:\n  line 1: cannot unmarshal !!int `50694` into subprocess.Process"
Oct 22 20:21:35 ns3171943 lxd.daemon[50137]: t=2020-10-22T20:21:35+0100 lvl=eror msg="Failed starting container" action=start created=2020-08-26T10:12:26+0100 ephemeral=false name=bc-crate-1ae3b69d-b903-4fd3-9c63-cbf940d7f8df project=default stateful=false used=2020-10-22T20:20:31+0100
Oct 22 20:27:28 ns3171943 lxd.daemon[50137]: t=2020-10-22T20:27:28+0100 lvl=warn msg="Detected poll(POLLNVAL) event."
Oct 22 20:29:20 ns3171943 lxd.daemon[50137]: t=2020-10-22T20:29:20+0100 lvl=warn msg="Detected poll(POLLNVAL) event."
Oct 22 20:31:27 ns3171943 lxd.daemon[50137]: t=2020-10-22T20:31:27+0100 lvl=warn msg="Detected poll(POLLNVAL) event."
Oct 22 20:33:17 ns3171943 lxd.daemon[50137]: t=2020-10-22T20:33:17+0100 lvl=warn msg="Detected poll(POLLNVAL) event."
Oct 22 20:34:48 ns3171943 lxd.daemon[50137]: t=2020-10-22T20:34:48+0100 lvl=warn msg="Detected poll(POLLNVAL) event."
Oct 22 20:36:04 ns3171943 lxd.daemon[50137]: t=2020-10-22T20:36:04+0100 lvl=warn msg="Detected poll(POLLNVAL) event."
Oct 22 20:49:47 ns3171943 lxd.daemon[50137]: t=2020-10-22T20:49:47+0100 lvl=warn msg="Detected poll(POLLNVAL) event."
Oct 22 20:52:05 ns3171943 lxd.daemon[50137]: t=2020-10-22T20:52:05+0100 lvl=warn msg="Detected poll(POLLNVAL) event."
Oct 22 20:53:18 ns3171943 lxd.daemon[50137]: t=2020-10-22T20:53:18+0100 lvl=warn msg="Detected poll(POLLNVAL) event."
Oct 23 10:51:03 ns3171943 lxd.daemon[50137]: t=2020-10-23T10:51:03+0100 lvl=warn msg="Detected poll(POLLNVAL) event."
Oct 26 10:44:49 ns3171943 lxd.daemon[50137]: t=2020-10-26T10:44:49+0000 lvl=eror msg="The stop hook failed" container=bc-crate-1d0aa532-add0-40b9-a7a8-b44689be5824 err="Failed to unmount '/var/snap/lxd/common/lxd/storage-pools/master/containers/bc-crate-1d0aa532-add0-40b9-a7a8-b44689be5824': device or resource busy"
Oct 26 16:42:17 ns3171943 systemd[1]: Stopping Service for snap application lxd.daemon...
Oct 26 16:42:17 ns3171943 lxd.daemon[669145]: => Stop reason is: snap refresh
Oct 26 16:42:17 ns3171943 lxd.daemon[669145]: => Stopping LXD
Oct 26 16:47:17 ns3171943 lxd.daemon[50007]: => LXD exited cleanly
Oct 26 16:47:18 ns3171943 lxd.daemon[669145]: ==> Stopped LXD
Oct 26 16:47:18 ns3171943 systemd[1]: snap.lxd.daemon.service: Succeeded.
Oct 26 16:47:18 ns3171943 systemd[1]: Stopped Service for snap application lxd.daemon.
Oct 26 16:47:33 ns3171943 systemd[1]: Started Service for snap application lxd.daemon.
Oct 26 16:47:33 ns3171943 lxd.daemon[675922]: => Preparing the system (18013)
Oct 26 16:47:33 ns3171943 lxd.daemon[675922]: ==> Loading snap configuration
Oct 26 16:47:33 ns3171943 lxd.daemon[675922]: ==> Setting up mntns symlink (mnt:[4026532681])
Oct 26 16:47:33 ns3171943 lxd.daemon[675922]: ==> Setting up kmod wrapper
Oct 26 16:47:33 ns3171943 lxd.daemon[675922]: ==> Preparing /boot
Oct 26 16:47:33 ns3171943 lxd.daemon[675922]: ==> Preparing a clean copy of /run
Oct 26 16:47:33 ns3171943 lxd.daemon[675922]: ==> Preparing /run/bin
Oct 26 16:47:33 ns3171943 lxd.daemon[675922]: ==> Preparing a clean copy of /etc
Oct 26 16:47:33 ns3171943 lxd.daemon[675922]: ==> Preparing a clean copy of /usr/share/misc
Oct 26 16:47:33 ns3171943 lxd.daemon[675922]: ==> Setting up ceph configuration
Oct 26 16:47:33 ns3171943 lxd.daemon[675922]: ==> Setting up LVM configuration
Oct 26 16:47:33 ns3171943 lxd.daemon[675922]: ==> Rotating logs
Oct 26 16:47:33 ns3171943 lxd.daemon[675922]: ==> Setting up ZFS (0.8)
Oct 26 16:47:33 ns3171943 lxd.daemon[675922]: ==> Escaping the systemd cgroups
Oct 26 16:47:33 ns3171943 lxd.daemon[675922]: ====> Detected cgroup V1
Oct 26 16:47:33 ns3171943 lxd.daemon[675922]: ==> Escaping the systemd process resource limits
Oct 26 16:47:33 ns3171943 lxd.daemon[675922]: ==> Disabling shiftfs on this kernel (auto)
Oct 26 16:47:33 ns3171943 lxd.daemon[675922]: ==> Detected kernel with partial AppArmor support
Oct 26 16:47:33 ns3171943 lxd.daemon[675922]: => Re-using existing LXCFS
Oct 26 16:47:33 ns3171943 lxd.daemon[675922]: => Starting LXD
Oct 26 16:47:33 ns3171943 lxd.daemon[676047]: t=2020-10-26T16:47:33+0000 lvl=warn msg=" - Couldn't find the CGroup blkio.weight, I/O weight limits will be ignored"
Oct 26 16:47:33 ns3171943 lxd.daemon[676047]: t=2020-10-26T16:47:33+0000 lvl=warn msg=" - Couldn't find the CGroup hugetlb controller, hugepage limits will be ignored"
Oct 26 16:47:33 ns3171943 lxd.daemon[676047]: t=2020-10-26T16:47:33+0000 lvl=warn msg=" - Couldn't find the CGroup memory swap accounting, swap limits will be ignored"
Oct 26 16:47:35 ns3171943 lxd.daemon[50126]: Closed liblxcfs.so
Oct 26 16:47:35 ns3171943 lxd.daemon[50126]: Running destructor lxcfs_exit
Oct 26 16:47:35 ns3171943 lxd.daemon[50126]: Running constructor lxcfs_init to reload liblxcfs
Oct 26 16:47:35 ns3171943 lxd.daemon[50126]: mount namespace: 5
Oct 26 16:47:35 ns3171943 lxd.daemon[50126]: hierarchies:
Oct 26 16:47:35 ns3171943 lxd.daemon[50126]:   0: fd:   6:
Oct 26 16:47:35 ns3171943 lxd.daemon[50126]:   1: fd:   7: name=systemd
Oct 26 16:47:35 ns3171943 lxd.daemon[50126]:   2: fd:   8: freezer
Oct 26 16:47:35 ns3171943 lxd.daemon[50126]:   3: fd:   9: cpu,cpuacct
Oct 26 16:47:35 ns3171943 lxd.daemon[50126]:   4: fd:  10: devices
Oct 26 16:47:35 ns3171943 lxd.daemon[50126]:   5: fd:  11: perf_event
Oct 26 16:47:35 ns3171943 lxd.daemon[50126]:   6: fd:  12: cpuset
Oct 26 16:47:35 ns3171943 lxd.daemon[50126]:   7: fd:  13: net_cls,net_prio
Oct 26 16:47:35 ns3171943 lxd.daemon[50126]:   8: fd:  14: blkio
Oct 26 16:47:35 ns3171943 lxd.daemon[50126]:   9: fd:  15: rdma
Oct 26 16:47:35 ns3171943 lxd.daemon[50126]:  10: fd:  16: pids
Oct 26 16:47:35 ns3171943 lxd.daemon[50126]:  11: fd:  18: memory
Oct 26 16:47:35 ns3171943 lxd.daemon[50126]: Kernel supports pidfds
Oct 26 16:47:35 ns3171943 lxd.daemon[50126]: Kernel does not support swap accounting
Oct 26 16:47:35 ns3171943 lxd.daemon[50126]: api_extensions:
Oct 26 16:47:35 ns3171943 lxd.daemon[50126]: - cgroups
Oct 26 16:47:35 ns3171943 lxd.daemon[50126]: - sys_cpu_online
Oct 26 16:47:35 ns3171943 lxd.daemon[50126]: - proc_cpuinfo
Oct 26 16:47:35 ns3171943 lxd.daemon[50126]: - proc_diskstats
Oct 26 16:47:35 ns3171943 lxd.daemon[50126]: - proc_loadavg
Oct 26 16:47:35 ns3171943 lxd.daemon[50126]: - proc_meminfo
Oct 26 16:47:35 ns3171943 lxd.daemon[50126]: - proc_stat
Oct 26 16:47:35 ns3171943 lxd.daemon[50126]: - proc_swaps
Oct 26 16:47:35 ns3171943 lxd.daemon[50126]: - proc_uptime
Oct 26 16:47:35 ns3171943 lxd.daemon[50126]: - shared_pidns
Oct 26 16:47:35 ns3171943 lxd.daemon[50126]: - cpuview_daemon
Oct 26 16:47:35 ns3171943 lxd.daemon[50126]: - loadavg_daemon
Oct 26 16:47:35 ns3171943 lxd.daemon[50126]: - pidfds
Oct 26 16:47:35 ns3171943 lxd.daemon[50126]: Reloaded LXCFS
Oct 26 16:47:36 ns3171943 lxd.daemon[675922]: => LXD is ready
Oct 29 19:08:12 ns3171943 lxd.daemon[676047]: t=2020-10-29T19:08:12+0000 lvl=warn msg="Detected poll(POLLNVAL) event: exiting."
Oct 29 19:08:12 ns3171943 lxd.daemon[676047]: t=2020-10-29T19:08:12+0000 lvl=warn msg="Detected poll(POLLNVAL) event."
Oct 30 05:37:15 ns3171943 systemd[1]: Stopping Service for snap application lxd.daemon...
Oct 30 05:37:16 ns3171943 lxd.daemon[1854435]: => Stop reason is: snap refresh
Oct 30 05:37:16 ns3171943 lxd.daemon[1854435]: => Stopping LXD
Oct 30 05:42:16 ns3171943 lxd.daemon[675922]: => LXD exited cleanly
Oct 30 05:42:16 ns3171943 lxd.daemon[1854435]: ==> Stopped LXD
Oct 30 05:42:16 ns3171943 systemd[1]: snap.lxd.daemon.service: Succeeded.
Oct 30 05:42:16 ns3171943 systemd[1]: Stopped Service for snap application lxd.daemon.
Oct 30 05:42:30 ns3171943 systemd[1]: Started Service for snap application lxd.daemon.
Oct 30 05:42:30 ns3171943 lxd.daemon[1860027]: => Preparing the system (18077)
Oct 30 05:42:30 ns3171943 lxd.daemon[1860058]: cmd_linux.go:160: cannot read /proc/self/exe: readlink /proc/self/exe: no such file or directory
Oct 30 05:42:30 ns3171943 lxd.daemon[1860027]: ==> Loading snap configuration
Oct 30 05:42:30 ns3171943 lxd.daemon[1860027]: ==> Setting up mntns symlink (mnt:[4026533157])
Oct 30 05:42:30 ns3171943 lxd.daemon[1860027]: ==> Setting up mount propagation on /var/snap/lxd/common/lxd/storage-pools
Oct 30 05:42:30 ns3171943 lxd.daemon[1860027]: ==> Setting up mount propagation on /var/snap/lxd/common/lxd/devices
Oct 30 05:42:30 ns3171943 lxd.daemon[1860027]: ==> Setting up persistent shmounts path
Oct 30 05:42:30 ns3171943 lxd.daemon[1860027]: ====> Making LXD shmounts use the persistent path
Oct 30 05:42:30 ns3171943 lxd.daemon[1860027]: ====> Making LXCFS use the persistent path
Oct 30 05:42:30 ns3171943 lxd.daemon[1860027]: ==> Setting up kmod wrapper
Oct 30 05:42:30 ns3171943 lxd.daemon[1860027]: ==> Preparing /boot
Oct 30 05:42:30 ns3171943 lxd.daemon[1860027]: ==> Preparing a clean copy of /run
Oct 30 05:42:30 ns3171943 lxd.daemon[1860027]: ==> Preparing /run/bin
Oct 30 05:42:30 ns3171943 lxd.daemon[1860027]: ==> Preparing a clean copy of /etc
Oct 30 05:42:30 ns3171943 lxd.daemon[1860027]: ==> Preparing a clean copy of /usr/share/misc
Oct 30 05:42:30 ns3171943 lxd.daemon[1860027]: ==> Setting up ceph configuration
Oct 30 05:42:30 ns3171943 lxd.daemon[1860027]: ==> Setting up LVM configuration
Oct 30 05:42:30 ns3171943 lxd.daemon[1860027]: ==> Rotating logs
Oct 30 05:42:30 ns3171943 lxd.daemon[1860027]: ==> Setting up ZFS (0.8)
Oct 30 05:42:30 ns3171943 lxd.daemon[1860027]: ==> Escaping the systemd cgroups
Oct 30 05:42:30 ns3171943 lxd.daemon[1860027]: ====> Detected cgroup V1
Oct 30 05:42:30 ns3171943 lxd.daemon[1860027]: ==> Escaping the systemd process resource limits
Oct 30 05:42:30 ns3171943 lxd.daemon[1860027]: ==> Disabling shiftfs on this kernel (auto)
Oct 30 05:42:30 ns3171943 lxd.daemon[1860027]: ==> Detected kernel with partial AppArmor support
Oct 30 05:42:30 ns3171943 lxd.daemon[1860027]: => Starting LXCFS
Oct 30 05:42:30 ns3171943 lxd.daemon[1860160]: Running constructor lxcfs_init to reload liblxcfs
Oct 30 05:42:30 ns3171943 lxd.daemon[1860160]: mount namespace: 4
Oct 30 05:42:30 ns3171943 lxd.daemon[1860160]: hierarchies:
Oct 30 05:42:30 ns3171943 lxd.daemon[1860160]:   0: fd:   5:
Oct 30 05:42:30 ns3171943 lxd.daemon[1860160]:   1: fd:   6: name=systemd
Oct 30 05:42:30 ns3171943 lxd.daemon[1860160]:   2: fd:   7: freezer
Oct 30 05:42:30 ns3171943 lxd.daemon[1860160]:   3: fd:   8: cpu,cpuacct
Oct 30 05:42:30 ns3171943 lxd.daemon[1860160]:   4: fd:   9: devices
Oct 30 05:42:30 ns3171943 lxd.daemon[1860160]:   5: fd:  10: perf_event
Oct 30 05:42:30 ns3171943 lxd.daemon[1860160]:   6: fd:  11: cpuset
Oct 30 05:42:30 ns3171943 lxd.daemon[1860160]:   7: fd:  12: net_cls,net_prio
Oct 30 05:42:30 ns3171943 lxd.daemon[1860160]:   8: fd:  13: blkio
Oct 30 05:42:30 ns3171943 lxd.daemon[1860160]:   9: fd:  14: rdma
Oct 30 05:42:30 ns3171943 lxd.daemon[1860160]:  10: fd:  15: pids
Oct 30 05:42:30 ns3171943 lxd.daemon[1860160]:  11: fd:  16: memory
Oct 30 05:42:30 ns3171943 lxd.daemon[1860160]: Kernel supports pidfds
Oct 30 05:42:30 ns3171943 lxd.daemon[1860160]: Kernel does not support swap accounting
Oct 30 05:42:30 ns3171943 lxd.daemon[1860160]: api_extensions:
Oct 30 05:42:30 ns3171943 lxd.daemon[1860160]: - cgroups
Oct 30 05:42:30 ns3171943 lxd.daemon[1860160]: - sys_cpu_online
Oct 30 05:42:30 ns3171943 lxd.daemon[1860160]: - proc_cpuinfo
Oct 30 05:42:30 ns3171943 lxd.daemon[1860160]: - proc_diskstats
Oct 30 05:42:30 ns3171943 lxd.daemon[1860160]: - proc_loadavg
Oct 30 05:42:30 ns3171943 lxd.daemon[1860160]: - proc_meminfo
Oct 30 05:42:30 ns3171943 lxd.daemon[1860160]: - proc_stat
Oct 30 05:42:30 ns3171943 lxd.daemon[1860160]: - proc_swaps
Oct 30 05:42:30 ns3171943 lxd.daemon[1860160]: - proc_uptime
Oct 30 05:42:30 ns3171943 lxd.daemon[1860160]: - shared_pidns
Oct 30 05:42:30 ns3171943 lxd.daemon[1860160]: - cpuview_daemon
Oct 30 05:42:30 ns3171943 lxd.daemon[1860160]: - loadavg_daemon
Oct 30 05:42:30 ns3171943 lxd.daemon[1860160]: - pidfds
Oct 30 05:42:31 ns3171943 lxd.daemon[1860027]: => Starting LXD
Oct 30 05:42:31 ns3171943 lxd.daemon[1860172]: t=2020-10-30T05:42:31+0000 lvl=warn msg=" - Couldn't find the CGroup blkio.weight, I/O weight limits will be ignored"
Oct 30 05:42:31 ns3171943 lxd.daemon[1860172]: t=2020-10-30T05:42:31+0000 lvl=warn msg=" - Couldn't find the CGroup hugetlb controller, hugepage limits will be ignored"
Oct 30 05:42:31 ns3171943 lxd.daemon[1860172]: t=2020-10-30T05:42:31+0000 lvl=warn msg=" - Couldn't find the CGroup memory swap accounting, swap limits will be ignored"
Oct 30 05:42:33 ns3171943 lxd.daemon[1860027]: => LXD is ready
Nov 03 00:27:17 ns3171943 systemd[1]: Stopping Service for snap application lxd.daemon...
Nov 03 00:27:17 ns3171943 lxd.daemon[3729664]: => Stop reason is: snap refresh
Nov 03 00:27:17 ns3171943 lxd.daemon[3729664]: => Stopping LXD
Nov 03 00:32:17 ns3171943 lxd.daemon[1860027]: => LXD exited cleanly
Nov 03 00:32:18 ns3171943 lxd.daemon[3729664]: ==> Stopped LXD
Nov 03 00:32:18 ns3171943 systemd[1]: snap.lxd.daemon.service: Succeeded.
Nov 03 00:32:18 ns3171943 systemd[1]: Stopped Service for snap application lxd.daemon.
Nov 03 00:32:38 ns3171943 systemd[1]: Started Service for snap application lxd.daemon.
Nov 03 00:32:38 ns3171943 lxd.daemon[3737276]: => Preparing the system (18137)
Nov 03 00:32:38 ns3171943 lxd.daemon[3737276]: ==> Loading snap configuration
Nov 03 00:32:38 ns3171943 lxd.daemon[3737276]: ==> Setting up mntns symlink (mnt:[4026533157])
Nov 03 00:32:38 ns3171943 lxd.daemon[3737276]: ==> Setting up kmod wrapper
Nov 03 00:32:38 ns3171943 lxd.daemon[3737276]: ==> Preparing /boot
Nov 03 00:32:38 ns3171943 lxd.daemon[3737276]: ==> Preparing a clean copy of /run
Nov 03 00:32:38 ns3171943 lxd.daemon[3737276]: ==> Preparing /run/bin
Nov 03 00:32:38 ns3171943 lxd.daemon[3737276]: ==> Preparing a clean copy of /etc
Nov 03 00:32:38 ns3171943 lxd.daemon[3737276]: ==> Preparing a clean copy of /usr/share/misc
Nov 03 00:32:38 ns3171943 lxd.daemon[3737276]: ==> Setting up ceph configuration
Nov 03 00:32:38 ns3171943 lxd.daemon[3737276]: ==> Setting up LVM configuration
Nov 03 00:32:38 ns3171943 lxd.daemon[3737276]: ==> Rotating logs
Nov 03 00:32:38 ns3171943 lxd.daemon[3737276]: ==> Setting up ZFS (0.8)
Nov 03 00:32:38 ns3171943 lxd.daemon[3737276]: ==> Escaping the systemd cgroups
Nov 03 00:32:38 ns3171943 lxd.daemon[3737276]: ====> Detected cgroup V1
Nov 03 00:32:38 ns3171943 lxd.daemon[3737276]: ==> Escaping the systemd process resource limits
Nov 03 00:32:38 ns3171943 lxd.daemon[3737276]: ==> Disabling shiftfs on this kernel (auto)
Nov 03 00:32:38 ns3171943 lxd.daemon[3737276]: ==> Detected kernel with partial AppArmor support
Nov 03 00:32:38 ns3171943 lxd.daemon[3737276]: => Re-using existing LXCFS
Nov 03 00:32:38 ns3171943 lxd.daemon[3737276]: => Starting LXD
Nov 03 00:32:38 ns3171943 lxd.daemon[3737399]: t=2020-11-03T00:32:38+0000 lvl=warn msg=" - Couldn't find the CGroup blkio.weight, I/O weight limits will be ignored"
Nov 03 00:32:38 ns3171943 lxd.daemon[3737399]: t=2020-11-03T00:32:38+0000 lvl=warn msg=" - Couldn't find the CGroup hugetlb controller, hugepage limits will be ignored"
Nov 03 00:32:38 ns3171943 lxd.daemon[3737399]: t=2020-11-03T00:32:38+0000 lvl=warn msg=" - Couldn't find the CGroup memory swap accounting, swap limits will be ignored"
Nov 03 00:32:40 ns3171943 lxd.daemon[3737276]: => LXD is ready
Nov 03 15:12:41 ns3171943 lxd.daemon[1860160]: Closed liblxcfs.so
Nov 03 15:12:41 ns3171943 lxd.daemon[1860160]: Running destructor lxcfs_exit
Nov 03 15:12:41 ns3171943 lxd.daemon[1860160]: Running constructor lxcfs_init to reload liblxcfs
Nov 03 15:12:41 ns3171943 lxd.daemon[1860160]: mount namespace: 5
Nov 03 15:12:41 ns3171943 lxd.daemon[1860160]: hierarchies:
Nov 03 15:12:41 ns3171943 lxd.daemon[1860160]:   0: fd:   6:
Nov 03 15:12:41 ns3171943 lxd.daemon[1860160]:   1: fd:   7: name=systemd
Nov 03 15:12:41 ns3171943 lxd.daemon[1860160]:   2: fd:   8: freezer
Nov 03 15:12:41 ns3171943 lxd.daemon[1860160]:   3: fd:   9: cpu,cpuacct
Nov 03 15:12:41 ns3171943 lxd.daemon[1860160]:   4: fd:  10: devices
Nov 03 15:12:41 ns3171943 lxd.daemon[1860160]:   5: fd:  11: perf_event
Nov 03 15:12:41 ns3171943 lxd.daemon[1860160]:   6: fd:  12: cpuset
Nov 03 15:12:41 ns3171943 lxd.daemon[1860160]:   7: fd:  13: net_cls,net_prio
Nov 03 15:12:41 ns3171943 lxd.daemon[1860160]:   8: fd:  14: blkio
Nov 03 15:12:41 ns3171943 lxd.daemon[1860160]:   9: fd:  15: rdma
Nov 03 15:12:41 ns3171943 lxd.daemon[1860160]:  10: fd:  16: pids
Nov 03 15:12:41 ns3171943 lxd.daemon[1860160]:  11: fd:  18: memory
Nov 03 15:12:41 ns3171943 lxd.daemon[1860160]: Kernel supports pidfds
Nov 03 15:12:41 ns3171943 lxd.daemon[1860160]: Kernel does not support swap accounting
Nov 03 15:12:41 ns3171943 lxd.daemon[1860160]: api_extensions:
Nov 03 15:12:41 ns3171943 lxd.daemon[1860160]: - cgroups
Nov 03 15:12:41 ns3171943 lxd.daemon[1860160]: - sys_cpu_online
Nov 03 15:12:41 ns3171943 lxd.daemon[1860160]: - proc_cpuinfo
Nov 03 15:12:41 ns3171943 lxd.daemon[1860160]: - proc_diskstats
Nov 03 15:12:41 ns3171943 lxd.daemon[1860160]: - proc_loadavg
Nov 03 15:12:41 ns3171943 lxd.daemon[1860160]: - proc_meminfo
Nov 03 15:12:41 ns3171943 lxd.daemon[1860160]: - proc_stat
Nov 03 15:12:41 ns3171943 lxd.daemon[1860160]: - proc_swaps
Nov 03 15:12:41 ns3171943 lxd.daemon[1860160]: - proc_uptime
Nov 03 15:12:41 ns3171943 lxd.daemon[1860160]: - shared_pidns
Nov 03 15:12:41 ns3171943 lxd.daemon[1860160]: - cpuview_daemon
Nov 03 15:12:41 ns3171943 lxd.daemon[1860160]: - loadavg_daemon
Nov 03 15:12:41 ns3171943 lxd.daemon[1860160]: - pidfds
Nov 03 15:12:41 ns3171943 lxd.daemon[1860160]: Reloaded LXCFS

Anything else I can add to this thread that might help diagnose the problem?

Something happened on Oct 30 at 05:42am which caused the shared mount to be reset, it’s unclear what though.

I don’t suppose snap changes shows anything useful as that was quite a while ago now.

Unfortunetly nothing seems to be coming up.

$ snap changes
error: no changes found

I’ve kept the container running on my system and still fails to delete incase any commands can be run to further diagnose what’s going wrong.

We’re unlikely to get anything more at this stage. It would have been useful to know if a core snap change or something similar happened around the time the issue showed up on the 30th but since snapd doesn’t have a full history, we’re unlikely to know until it somehow happens again…