Lxd crashing constantly

I have lxd installed as a snap. It crashes and thus I can’t connect from lxc, so it’s unusable. It’s just repeatedly doing this.

Dec 21 19:27:27 hal systemd[1]: snap.lxd.daemon.service: Service hold-off time over, scheduling restart.
Dec 21 19:27:27 hal systemd[1]: Stopped Service for snap application lxd.daemon.
Dec 21 19:27:27 hal systemd[1]: Started Service for snap application lxd.daemon.
Dec 21 19:27:27 hal lxd.daemon[873]: => Preparing the system
Dec 21 19:27:27 hal lxd.daemon[873]: ==> Loading snap configuration
Dec 21 19:27:27 hal lxd.daemon[873]: ==> Setting up modprobe wrapper
Dec 21 19:27:27 hal lxd.daemon[873]: ==> Preparing a clean copy of /run
Dec 21 19:27:27 hal lxd.daemon[873]: ==> Preparing a clean copy of /etc
Dec 21 19:27:27 hal lxd.daemon[873]: ==> Setting up ceph configuration
Dec 21 19:27:27 hal lxd.daemon[873]: ==> Setting up LVM configuration
Dec 21 19:27:27 hal lxd.daemon[873]: ==> Escaping the systemd cgroups
Dec 21 19:27:27 hal lxd.daemon[873]: ==> Escaping the systemd process resource limits
Dec 21 19:27:27 hal lxd.daemon[873]: ==> Enabling unprivileged containers kernel support
Dec 21 19:27:27 hal lxd.daemon[5652]: hierarchies:
Dec 21 19:27:27 hal lxd.daemon[5652]:   0: fd:   7: freezer
Dec 21 19:27:27 hal lxd.daemon[5652]:   1: fd:   8: memory
Dec 21 19:27:27 hal lxd.daemon[5652]:   2: fd:   9: perf_event
Dec 21 19:27:27 hal lxd.daemon[5652]:   3: fd:  10: devices
Dec 21 19:27:27 hal lxd.daemon[5652]:   4: fd:  11: pids
Dec 21 19:27:27 hal lxd.daemon[5652]:   5: fd:  12: cpuset
Dec 21 19:27:27 hal lxd.daemon[5652]:   6: fd:  13: net_cls,net_prio
Dec 21 19:27:27 hal lxd.daemon[5652]:   7: fd:  14: hugetlb
Dec 21 19:27:27 hal lxd.daemon[5652]:   8: fd:  15: cpu,cpuacct
Dec 21 19:27:27 hal lxd.daemon[5652]:   9: fd:  16: blkio
Dec 21 19:27:27 hal lxd.daemon[5652]:  10: fd:  17: name=systemd
Dec 21 19:27:27 hal lxd.daemon[5652]: lxcfs.c: 105: do_reload: lxcfs: reloaded
Dec 21 19:27:27 hal lxd.daemon[873]: => Re-using existing LXCFS
Dec 21 19:27:27 hal lxd.daemon[873]: => Starting LXD
Dec 21 19:27:27 hal lxd.daemon[873]: lvl=warn msg="CGroup memory swap accounting is disabled, swap limits will be ignored." t=2017-12-21T19:27:27+0000
Dec 21 19:27:27 hal lxd.daemon[873]: lvl=eror msg="Failed to rename device from \"/var/snap/lxd/common/lxd/devices/snapcraft-josm/disk.root-build_josm\" to \"/var/snap/lxd/common/lxd/devices/snapcraft-josm/disk./root/build_josm.root-build_josm\": rename /var/snap/lxd/common/lxd/devices/snapcraft-josm/disk.root-build_josm /var/snap/lxd/common/lxd/devices/snapcraft-josm/disk./root/build_josm.root-build_josm: no such file or directory" t=2017-12-21T19:27:27+0000
Dec 21 19:27:27 hal lxd.daemon[873]: error: rename /var/snap/lxd/common/lxd/devices/snapcraft-josm/disk.root-build_josm /var/snap/lxd/common/lxd/devices/snapcraft-josm/disk./root/build_josm.root-build_josm: no such file or directory
Dec 21 19:37:27 hal lxd.daemon[873]: error: LXD still not running after 600s timeout.
Dec 21 19:37:27 hal systemd[1]: snap.lxd.daemon.service: Main process exited, code=exited, status=1/FAILURE
Dec 21 19:37:27 hal lxd.daemon[3020]: => Stop reason is: crashed
Dec 21 19:37:27 hal systemd[1]: snap.lxd.daemon.service: Unit entered failed state.
Dec 21 19:37:27 hal systemd[1]: snap.lxd.daemon.service: Failed with result 'exit-code'.
alan@hal:~$ snap version
snap    2.30
snapd   2.30
series  16
ubuntu  16.04
kernel  4.10.0-42-generic
name:      lxd
summary:   System container manager and API
publisher: canonical
contact:   https://github.com/lxc/lxd/issues
description: |
  LXD is a container manager for system containers.
  
  It offers a REST API to remotely manage containers over the network, using an
  image based workflow and with support for live migration.
  
  Images are available for all Ubuntu releases and architectures as well as for
  a wide number of other Linux distributions.
  
  LXD containers are lightweight, secure by default and a great alternative to
  virtual machines.
snap-id: J60k4JY0HppjwOjW8dZdYc8obXKxujRu
commands:
  - lxd.benchmark
  - lxd.check-kernel
  - lxd.lxc
  - lxd
  - lxd.migrate
services:
  lxd.daemon:    simple, enabled, active
tracking:        stable
installed:       2.21 (5373) 44MB -
refreshed:       2017-12-21 03:30:04 +0000 UTC
channels:                           
  stable:        2.21        (5373) 44MB -
  candidate:     2.21        (5373) 44MB -
  beta:          ↑                       
  edge:          git-e0b0f09 (5392) 44MB -
  2.0/stable:    2.0.11      (5384) 21MB -
  2.0/candidate: 2.0.11      (5384) 21MB -
  2.0/beta:      ↑                       
  2.0/edge:      git-3931f6d (5379) 20MB -

@brauner

   Dec 21 19:27:27 hal lxd.daemon[873]: error: rename /var/snap/lxd/common/lxd/devices/snapcraft-josm/disk.root-build_josm /var/snap/lxd/common/lxd/devices/snapcraft-josm/disk./root/build_josm.root-build_josm: no such file or directory

Looks like something bad’s going on with the upgrade logic.

@popey can you get me the output of:

ls -lh /var/snap/lxd/common/lxd/devices/
ls -lh /var/snap/lxd/common/lxd/devices/*/

@popey

Would also be very helpful if you could e-mail me your /var/snap/lxd/common/lxd/lxd.db file as that’d let me inspect your container’s configuration too and get you a quick fix for this.

Had to do this as root because my account had no permission:-

alan@hal:~$ ls -lh /var/snap/lxd/common/lxd/devices/
ls: cannot open directory '/var/snap/lxd/common/lxd/devices/': Permission denied
alan@hal:~$ ls -lh /var/snap/lxd/common/lxd/devices/*/
ls: cannot access '/var/snap/lxd/common/lxd/devices/*/': No such file or directory

root@hal:~# ls -lh /var/snap/lxd/common/lxd/devices/
total 56K
drwx--x--x 2 root root 4.0K Nov 24 16:57 cool-oarfish
drwx--x--x 2 root root 4.0K Dec 12 11:11 frank-satyr
drwx--x--x 2 root root 4.0K Nov 24 16:57 grateful-halibut
drwx--x--x 2 root root 4.0K Oct  9 15:03 snapcraft-elementary-os-sdk
drwx--x--x 2 root root 4.0K Dec 12 12:06 snapcraft-fetch-btc
drwx--x--x 2 root root 4.0K Nov 20 13:11 snapcraft-infinitypfm
drwx--x--x 2 root root 4.0K Nov 20 15:06 snapcraft-jetty9
drwx--x--x 4 root root 4.0K Dec 22 15:55 snapcraft-josm
drwx--x--x 2 root root 4.0K Dec 15 22:22 snapcraft-minecraft
drwx--x--x 2 root root 4.0K Dec  8 14:06 snapcraft-onlyoffice
drwx--x--x 3 root root 4.0K Dec 14 09:33 snapcraft-steam
drwx--x--x 2 root root 4.0K Oct 12 15:09 snapcraft-wine
drwx--x--x 2 root root 4.0K Oct 25 22:22 unified-snapper
drwx--x--x 2 root root 4.0K Dec 21 00:23 wine
root@hal:~# ls -lh /var/snap/lxd/common/lxd/devices/*/
/var/snap/lxd/common/lxd/devices/cool-oarfish/:
total 0

/var/snap/lxd/common/lxd/devices/frank-satyr/:
total 0

/var/snap/lxd/common/lxd/devices/grateful-halibut/:
total 0

/var/snap/lxd/common/lxd/devices/snapcraft-elementary-os-sdk/:
total 0

/var/snap/lxd/common/lxd/devices/snapcraft-fetch-btc/:
total 0

/var/snap/lxd/common/lxd/devices/snapcraft-infinitypfm/:
total 0

/var/snap/lxd/common/lxd/devices/snapcraft-jetty9/:
total 0

/var/snap/lxd/common/lxd/devices/snapcraft-josm/:
total 8.0K
drwxr-xr-x 3 root root    4.0K Dec 22 15:54 disk.root.
drwx------ 2 root root    4.0K Dec 14 09:33 disk.root-build_josm
crw-rw-rw- 1 alan alan 10, 229 Dec 14 09:33 unix.dev-fuse

/var/snap/lxd/common/lxd/devices/snapcraft-minecraft/:
total 0

/var/snap/lxd/common/lxd/devices/snapcraft-onlyoffice/:
total 0

/var/snap/lxd/common/lxd/devices/snapcraft-steam/:
total 4.0K
drwx------ 2 root root 4.0K Dec 14 09:33 disk.root-build_steam

/var/snap/lxd/common/lxd/devices/snapcraft-wine/:
total 0

/var/snap/lxd/common/lxd/devices/unified-snapper/:
total 0

/var/snap/lxd/common/lxd/devices/wine/:
total 0
root@hal:~# 

Sent via email to your canonical account.

Thanks, that should let me reproduce this on a test system and get a fix out ASAP.
In the meantime, if you’re looking at getting back online as quickly as possible, you can do:

  • Reboot your system (that will ensure that all LXD containers and mounts are gone)
  • sudo rm -Rf /var/snap/lxd/common/lxd/devices//
  • sudo systemctl restart snap.lxd.daemon

That should get you past the upgrade code and back to a running LXD daemon.

Ah, I think the issue is that the device name itself contains a slash in your case. I’ll look at fixing this now.

stgraber@castiana:~$ lxc config device add snapcraft /dev/kvm unix-char path=/dev/kvm
error: Failed to setup device: Failed to create device /var/lib/lxd/devices/snapcraft/unix./dev/kvm.dev-kvm for /dev/kvm: no such file or directory

So yeah, that’s definitely the issue.

Fix cherry-picked in the snap (currently building in candidate channel) and will be released as soon as Jenkins gives the go ahead.

Same fix cherry-picked into the .deb too, will update the PPA and backports once it passes CI too.

Snap released to the stable channel now.

Still not happy…

alan@hal:~$ sudo rm -Rf /var/snap/lxd/common/lxd/devices/*/*
alan@hal:~$ sudo systemctl restart snap.lxd.daemon
alan@hal:~$ lxc list
LXD socket not found; is LXD installed and running?
alan@hal:~$ sudo snap refresh lxd
snap "lxd" has no updates available
alan@hal:~$ snap info lxd
name:      lxd
summary:   System container manager and API
publisher: canonical
contact:   https://github.com/lxc/lxd/issues
description: |
  LXD is a container manager for system containers.
  
  It offers a REST API to remotely manage containers over the network, using an
  image based workflow and with support for live migration.
  
  Images are available for all Ubuntu releases and architectures as well as for
  a wide number of other Linux distributions.
  
  LXD containers are lightweight, secure by default and a great alternative to
  virtual machines.
snap-id: J60k4JY0HppjwOjW8dZdYc8obXKxujRu
commands:
  - lxd.benchmark
  - lxd.check-kernel
  - lxd.lxc
  - lxd
  - lxd.migrate
services:
  lxd.daemon:    simple, enabled, active
tracking:        stable
installed:       2.21 (5397) 44MB -
refreshed:       2017-12-22 20:39:45 +0000 UTC
channels:                           
  stable:        2.21        (5397) 44MB -
  candidate:     2.21        (5408) 44MB -
  beta:          ↑                       
  edge:          git-efa955a (5406) 44MB -
  2.0/stable:    2.0.11      (5384) 21MB -
  2.0/candidate: 2.0.11      (5384) 21MB -
  2.0/beta:      ↑                       
  2.0/edge:      git-3931f6d (5379) 20MB -
You have new mail in /var/mail/alan
alan@hal:~$ lxc list
LXD socket not found; is LXD installed and running?

You may want to refresh again, another build just made it to stable (not that I’d expect it to change things…).

Once updated, if it still doesn’t work, can you show journalctl -u snap.lxd.daemon?

Thanks for the help. No, the latest snap didn’t fix it.

alan@hal:~$ journalctl -u snap.lxd.daemon
-- Logs begin at Fri 2017-12-22 21:48:05 GMT, end at Fri 2017-12-22 22:12:51 GMT
Dec 22 21:48:14 hal systemd[1]: Started Service for snap application lxd.daemon.
Dec 22 21:48:15 hal lxd.daemon[6196]: => Preparing the system
Dec 22 21:48:15 hal lxd.daemon[6196]: ==> Loading snap configuration
Dec 22 21:48:15 hal lxd.daemon[6196]: ==> Setting up modprobe wrapper
Dec 22 21:48:15 hal lxd.daemon[6196]: ==> Preparing a clean copy of /run
Dec 22 21:48:15 hal lxd.daemon[6196]: ==> Preparing a clean copy of /etc
Dec 22 21:48:15 hal lxd.daemon[6196]: ==> Setting up ceph configuration
Dec 22 21:48:15 hal lxd.daemon[6196]: ==> Setting up LVM configuration
Dec 22 21:48:15 hal lxd.daemon[6196]: ==> Escaping the systemd cgroups
Dec 22 21:48:15 hal lxd.daemon[6196]: ==> Escaping the systemd process resource 
Dec 22 21:48:15 hal lxd.daemon[6196]: ==> Increasing the number of inotify user 
Dec 22 21:48:15 hal lxd.daemon[6196]: ==> Enabling unprivileged containers kerne
Dec 22 21:48:15 hal lxd.daemon[6196]: => Starting LXCFS
Dec 22 21:48:15 hal lxd.daemon[6196]: => Starting LXD
Dec 22 21:48:15 hal lxd.daemon[6196]: hierarchies:
Dec 22 21:48:15 hal lxd.daemon[6196]:   0: fd:   5: devices
Dec 22 21:48:15 hal lxd.daemon[6196]:   1: fd:   6: freezer
Dec 22 21:48:15 hal lxd.daemon[6196]:   2: fd:   7: perf_event
Dec 22 21:48:15 hal lxd.daemon[6196]:   3: fd:   8: net_cls,net_prio
Dec 22 21:48:15 hal lxd.daemon[6196]:   4: fd:   9: blkio
Dec 22 21:48:15 hal lxd.daemon[6196]:   5: fd:  10: memory
Dec 22 21:48:15 hal lxd.daemon[6196]:   6: fd:  11: pids
Dec 22 21:48:15 hal lxd.daemon[6196]:   7: fd:  12: cpuset
Dec 22 21:48:15 hal lxd.daemon[6196]:   8: fd:  13: cpu,cpuacct
Dec 22 21:48:15 hal lxd.daemon[6196]:   9: fd:  14: hugetlb
Dec 22 21:48:15 hal lxd.daemon[6196]:  10: fd:  15: name=systemd
Dec 22 21:48:16 hal lxd.daemon[6196]: lvl=warn msg="CGroup memory swap accountin
Dec 22 21:48:16 hal lxd.daemon[6196]: lvl=eror msg="Failed to remove device \"/v
Dec 22 21:48:16 hal lxd.daemon[6196]: error: remove /var/snap/lxd/common/lxd/dev
Dec 22 21:48:24 hal systemd[1]: Stopping Service for snap application lxd.daemon
Dec 22 21:48:24 hal lxd.daemon[7273]: => Stop reason is: crashed
Dec 22 21:48:24 hal systemd[1]: Stopped Service for snap application lxd.daemon.
Dec 22 21:48:27 hal systemd[1]: Started Service for snap application lxd.daemon.
Dec 22 21:48:28 hal lxd.daemon[7538]: => Preparing the system
Dec 22 21:48:28 hal lxd.daemon[7538]: ==> Loading snap configuration
Dec 22 21:48:28 hal lxd.daemon[7538]: ==> Setting up modprobe wrapper
Dec 22 21:48:28 hal lxd.daemon[7538]: ==> Preparing a clean copy of /run
Dec 22 21:48:28 hal lxd.daemon[7538]: ==> Preparing a clean copy of /etc
Dec 22 21:48:28 hal lxd.daemon[7538]: ==> Setting up ceph configuration
Dec 22 21:48:28 hal lxd.daemon[7538]: ==> Setting up LVM configuration
Dec 22 21:48:28 hal lxd.daemon[7538]: ==> Escaping the systemd cgroups
Dec 22 21:48:28 hal lxd.daemon[7538]: ==> Escaping the systemd process resource 
Dec 22 21:48:28 hal lxd.daemon[7538]: ==> Enabling unprivileged containers kerne
Dec 22 21:48:28 hal lxd.daemon[7538]: => Re-using existing LXCFS
Dec 22 21:48:28 hal lxd.daemon[7538]: => Starting LXD
Dec 22 21:48:28 hal lxd.daemon[7538]: lvl=warn msg="CGroup memory swap accountin
Dec 22 21:48:28 hal lxd.daemon[7538]: lvl=eror msg="Failed to remove device \"/v
Dec 22 21:48:28 hal lxd.daemon[7538]: error: remove /var/snap/lxd/common/lxd/dev
Dec 22 21:49:12 hal systemd[1]: Stopping Service for snap application lxd.daemon
Dec 22 21:49:12 hal lxd.daemon[10364]: => Stop reason is: crashed
Dec 22 21:49:12 hal systemd[1]: Stopped Service for snap application lxd.daemon.
Dec 22 21:49:12 hal systemd[1]: Started Service for snap application lxd.daemon.
Dec 22 21:49:12 hal lxd.daemon[10395]: => Preparing the system
Dec 22 21:49:12 hal lxd.daemon[10395]: ==> Loading snap configuration
Dec 22 21:49:12 hal lxd.daemon[10395]: ==> Setting up modprobe wrapper
Dec 22 21:49:12 hal lxd.daemon[10395]: ==> Preparing a clean copy of /run
Dec 22 21:49:12 hal lxd.daemon[10395]: ==> Preparing a clean copy of /etc
Dec 22 21:49:12 hal lxd.daemon[10395]: ==> Setting up ceph configuration
Dec 22 21:49:12 hal lxd.daemon[10395]: ==> Setting up LVM configuration
Dec 22 21:49:12 hal lxd.daemon[10395]: ==> Escaping the systemd cgroups
Dec 22 21:49:12 hal lxd.daemon[10395]: ==> Escaping the systemd process resource
Dec 22 21:49:12 hal lxd.daemon[10395]: ==> Enabling unprivileged containers kern
Dec 22 21:49:12 hal lxd.daemon[6196]: hierarchies:
Dec 22 21:49:12 hal lxd.daemon[6196]:   0: fd:   7: devices
Dec 22 21:49:12 hal lxd.daemon[6196]:   1: fd:   8: freezer
Dec 22 21:49:12 hal lxd.daemon[6196]:   2: fd:   9: perf_event
Dec 22 21:49:12 hal lxd.daemon[6196]:   3: fd:  10: net_cls,net_prio
Dec 22 21:49:12 hal lxd.daemon[6196]:   4: fd:  11: blkio
Dec 22 21:49:12 hal lxd.daemon[6196]:   5: fd:  12: memory
Dec 22 21:49:12 hal lxd.daemon[6196]:   6: fd:  13: pids
Dec 22 21:49:12 hal lxd.daemon[6196]:   7: fd:  14: cpuset
Dec 22 21:49:12 hal lxd.daemon[6196]:   8: fd:  15: cpu,cpuacct
Dec 22 21:49:12 hal lxd.daemon[6196]:   9: fd:  16: hugetlb
Dec 22 21:49:12 hal lxd.daemon[6196]:  10: fd:  17: name=systemd
Dec 22 21:49:12 hal lxd.daemon[6196]: lxcfs.c: 105: do_reload: lxcfs: reloaded
Dec 22 21:49:12 hal lxd.daemon[10395]: => Re-using existing LXCFS
Dec 22 21:49:12 hal lxd.daemon[10395]: => Starting LXD
Dec 22 21:49:12 hal lxd.daemon[10395]: lvl=warn msg="CGroup memory swap accounti
Dec 22 21:49:12 hal lxd.daemon[10395]: lvl=eror msg="Failed to remove device \"/
Dec 22 21:49:12 hal lxd.daemon[10395]: error: remove /var/snap/lxd/common/lxd/de
Dec 22 21:58:16 hal lxd.daemon[6196]: error: LXD still not running after 600s ti
Dec 22 21:58:28 hal lxd.daemon[7538]: error: LXD still not running after 600s ti
Dec 22 21:59:12 hal lxd.daemon[10395]: error: LXD still not running after 600s t
Dec 22 21:59:12 hal systemd[1]: snap.lxd.daemon.service: Main process exited, co
Dec 22 21:59:12 hal lxd.daemon[10652]: => Stop reason is: crashed
Dec 22 21:59:12 hal systemd[1]: snap.lxd.daemon.service: Unit entered failed sta
Dec 22 21:59:12 hal systemd[1]: snap.lxd.daemon.service: Failed with result 'exi
Dec 22 21:59:13 hal systemd[1]: snap.lxd.daemon.service: Service hold-off time o
Dec 22 21:59:13 hal systemd[1]: Stopped Service for snap application lxd.daemon.
Dec 22 21:59:13 hal systemd[1]: Started Service for snap application lxd.daemon.
Dec 22 21:59:13 hal lxd.daemon[10684]: => Preparing the system
Dec 22 21:59:13 hal lxd.daemon[10684]: ==> Loading snap configuration
Dec 22 21:59:13 hal lxd.daemon[10684]: ==> Setting up modprobe wrapper
Dec 22 21:59:13 hal lxd.daemon[10684]: ==> Preparing a clean copy of /run
Dec 22 21:59:13 hal lxd.daemon[10684]: ==> Preparing a clean copy of /etc
Dec 22 21:59:13 hal lxd.daemon[10684]: ==> Setting up ceph configuration
Dec 22 21:59:13 hal lxd.daemon[10684]: ==> Setting up LVM configuration
Dec 22 21:59:13 hal lxd.daemon[10684]: ==> Escaping the systemd cgroups
Dec 22 21:59:13 hal lxd.daemon[10684]: ==> Escaping the systemd process resource
Dec 22 21:59:13 hal lxd.daemon[10684]: ==> Enabling unprivileged containers kern
Dec 22 21:59:13 hal lxd.daemon[6196]: hierarchies:
Dec 22 21:59:13 hal lxd.daemon[6196]:   0: fd:   7: devices
Dec 22 21:59:13 hal lxd.daemon[6196]:   1: fd:   8: freezer
Dec 22 21:59:13 hal lxd.daemon[6196]:   2: fd:   9: perf_event
Dec 22 21:59:13 hal lxd.daemon[6196]:   3: fd:  10: net_cls,net_prio
Dec 22 21:59:13 hal lxd.daemon[6196]:   4: fd:  11: blkio
Dec 22 21:59:13 hal lxd.daemon[6196]:   5: fd:  12: memory
Dec 22 21:59:13 hal lxd.daemon[6196]:   6: fd:  13: pids
Dec 22 21:59:13 hal lxd.daemon[6196]:   7: fd:  14: cpuset
Dec 22 21:59:13 hal lxd.daemon[6196]:   8: fd:  15: cpu,cpuacct
Dec 22 21:59:13 hal lxd.daemon[6196]:   9: fd:  16: hugetlb
Dec 22 21:59:13 hal lxd.daemon[6196]:  10: fd:  17: name=systemd
Dec 22 21:59:13 hal lxd.daemon[6196]: lxcfs.c: 105: do_reload: lxcfs: reloaded
Dec 22 21:59:13 hal lxd.daemon[10684]: => Re-using existing LXCFS
Dec 22 21:59:13 hal lxd.daemon[10684]: => Starting LXD
Dec 22 21:59:13 hal lxd.daemon[10684]: lvl=warn msg="CGroup memory swap accounti
Dec 22 21:59:13 hal lxd.daemon[10684]: lvl=eror msg="Failed to remove device \"/
Dec 22 21:59:13 hal lxd.daemon[10684]: error: remove /var/snap/lxd/common/lxd/de
Dec 22 22:09:13 hal lxd.daemon[10684]: error: LXD still not running after 600s t
Dec 22 22:09:13 hal systemd[1]: snap.lxd.daemon.service: Main process exited, co
Dec 22 22:09:13 hal lxd.daemon[10773]: => Stop reason is: crashed
Dec 22 22:09:13 hal systemd[1]: snap.lxd.daemon.service: Unit entered failed sta
Dec 22 22:09:13 hal systemd[1]: snap.lxd.daemon.service: Failed with result 'exi
Dec 22 22:09:13 hal systemd[1]: snap.lxd.daemon.service: Service hold-off time o
Dec 22 22:09:13 hal systemd[1]: Stopped Service for snap application lxd.daemon.
Dec 22 22:09:13 hal systemd[1]: Started Service for snap application lxd.daemon.
Dec 22 22:09:13 hal lxd.daemon[10806]: => Preparing the system
Dec 22 22:09:13 hal lxd.daemon[10806]: ==> Loading snap configuration
Dec 22 22:09:13 hal lxd.daemon[10806]: ==> Setting up modprobe wrapper
Dec 22 22:09:13 hal lxd.daemon[10806]: ==> Preparing a clean copy of /run
Dec 22 22:09:13 hal lxd.daemon[10806]: ==> Preparing a clean copy of /etc
Dec 22 22:09:13 hal lxd.daemon[10806]: ==> Setting up ceph configuration
Dec 22 22:09:13 hal lxd.daemon[10806]: ==> Setting up LVM configuration
Dec 22 22:09:13 hal lxd.daemon[10806]: ==> Escaping the systemd cgroups
Dec 22 22:09:13 hal lxd.daemon[10806]: ==> Escaping the systemd process resource
Dec 22 22:09:13 hal lxd.daemon[10806]: ==> Enabling unprivileged containers kern
Dec 22 22:09:13 hal lxd.daemon[6196]: hierarchies:
Dec 22 22:09:13 hal lxd.daemon[6196]:   0: fd:   7: devices
Dec 22 22:09:13 hal lxd.daemon[6196]:   1: fd:   8: freezer
Dec 22 22:09:13 hal lxd.daemon[6196]:   2: fd:   9: perf_event
Dec 22 22:09:13 hal lxd.daemon[6196]:   3: fd:  10: net_cls,net_prio
Dec 22 22:09:13 hal lxd.daemon[6196]:   4: fd:  11: blkio
Dec 22 22:09:13 hal lxd.daemon[6196]:   5: fd:  12: memory
Dec 22 22:09:13 hal lxd.daemon[6196]:   6: fd:  13: pids
Dec 22 22:09:13 hal lxd.daemon[6196]:   7: fd:  14: cpuset
Dec 22 22:09:13 hal lxd.daemon[6196]:   8: fd:  15: cpu,cpuacct
Dec 22 22:09:13 hal lxd.daemon[6196]:   9: fd:  16: hugetlb
Dec 22 22:09:13 hal lxd.daemon[6196]:  10: fd:  17: name=systemd
Dec 22 22:09:13 hal lxd.daemon[6196]: lxcfs.c: 105: do_reload: lxcfs: reloaded
Dec 22 22:09:13 hal lxd.daemon[10806]: => Re-using existing LXCFS
Dec 22 22:09:13 hal lxd.daemon[10806]: => Starting LXD
Dec 22 22:09:13 hal lxd.daemon[10806]: lvl=warn msg="CGroup memory swap accounti
Dec 22 22:09:13 hal lxd.daemon[10806]: lvl=eror msg="Failed to remove device \"/
Dec 22 22:09:13 hal lxd.daemon[10806]: error: remove /var/snap/lxd/common/lxd/de
Dec 22 22:12:28 hal systemd[1]: Stopping Service for snap application lxd.daemon
Dec 22 22:12:28 hal lxd.daemon[10994]: => Stop reason is: crashed
Dec 22 22:12:28 hal systemd[1]: Stopped Service for snap application lxd.daemon.
Dec 22 22:12:31 hal systemd[1]: Started Service for snap application lxd.daemon.
Dec 22 22:12:31 hal lxd.daemon[11159]: => Preparing the system
Dec 22 22:12:31 hal lxd.daemon[11159]: ==> Loading snap configuration
Dec 22 22:12:31 hal lxd.daemon[11159]: ==> Setting up modprobe wrapper
Dec 22 22:12:31 hal lxd.daemon[11159]: ==> Preparing a clean copy of /run
Dec 22 22:12:31 hal lxd.daemon[11159]: ==> Preparing a clean copy of /etc
Dec 22 22:12:31 hal lxd.daemon[11159]: ==> Setting up ceph configuration
Dec 22 22:12:31 hal lxd.daemon[11159]: ==> Setting up LVM configuration
Dec 22 22:12:31 hal lxd.daemon[11159]: ==> Escaping the systemd cgroups
Dec 22 22:12:31 hal lxd.daemon[11159]: ==> Escaping the systemd process resource
Dec 22 22:12:31 hal lxd.daemon[11159]: ==> Enabling unprivileged containers kern
Dec 22 22:12:31 hal lxd.daemon[6196]: hierarchies:
Dec 22 22:12:31 hal lxd.daemon[6196]:   0: fd:   7: devices
Dec 22 22:12:31 hal lxd.daemon[6196]:   1: fd:   8: freezer
Dec 22 22:12:31 hal lxd.daemon[6196]:   2: fd:   9: perf_event
Dec 22 22:12:31 hal lxd.daemon[6196]:   3: fd:  10: net_cls,net_prio
Dec 22 22:12:31 hal lxd.daemon[6196]:   4: fd:  11: blkio
Dec 22 22:12:31 hal lxd.daemon[6196]:   5: fd:  12: memory
Dec 22 22:12:31 hal lxd.daemon[6196]:   6: fd:  13: pids
Dec 22 22:12:31 hal lxd.daemon[6196]:   7: fd:  14: cpuset
Dec 22 22:12:31 hal lxd.daemon[6196]:   8: fd:  15: cpu,cpuacct
Dec 22 22:12:31 hal lxd.daemon[6196]:   9: fd:  16: hugetlb
Dec 22 22:12:31 hal lxd.daemon[6196]:  10: fd:  17: name=systemd
Dec 22 22:12:31 hal lxd.daemon[6196]: lxcfs.c: 105: do_reload: lxcfs: reloaded
Dec 22 22:12:31 hal lxd.daemon[11159]: => Re-using existing LXCFS
Dec 22 22:12:31 hal lxd.daemon[11159]: => Starting LXD
Dec 22 22:12:31 hal lxd.daemon[11159]: lvl=warn msg="CGroup memory swap accounti
Dec 22 22:12:32 hal lxd.daemon[11159]: lvl=eror msg="Failed to remove device \"/
Dec 22 22:12:32 hal lxd.daemon[11159]: error: remove /var/snap/lxd/common/lxd/de
Dec 22 22:12:50 hal systemd[1]: Stopping Service for snap application lxd.daemon
Dec 22 22:12:51 hal lxd.daemon[11325]: => Stop reason is: crashed
Dec 22 22:12:51 hal systemd[1]: Stopped Service for snap application lxd.daemon.
Dec 22 22:12:51 hal systemd[1]: Started Service for snap application lxd.daemon.
Dec 22 22:12:51 hal lxd.daemon[11357]: => Preparing the system
Dec 22 22:12:51 hal lxd.daemon[11357]: ==> Loading snap configuration
Dec 22 22:12:51 hal lxd.daemon[11357]: ==> Setting up modprobe wrapper
Dec 22 22:12:51 hal lxd.daemon[11357]: ==> Preparing a clean copy of /run
Dec 22 22:12:51 hal lxd.daemon[11357]: ==> Preparing a clean copy of /etc
Dec 22 22:12:51 hal lxd.daemon[11357]: ==> Setting up ceph configuration
Dec 22 22:12:51 hal lxd.daemon[11357]: ==> Setting up LVM configuration
Dec 22 22:12:51 hal lxd.daemon[11357]: ==> Escaping the systemd cgroups
Dec 22 22:12:51 hal lxd.daemon[11357]: ==> Escaping the systemd process resource
Dec 22 22:12:51 hal lxd.daemon[11357]: ==> Enabling unprivileged containers kern
Dec 22 22:12:51 hal lxd.daemon[6196]: hierarchies:
Dec 22 22:12:51 hal lxd.daemon[6196]:   0: fd:   7: devices
Dec 22 22:12:51 hal lxd.daemon[6196]:   1: fd:   8: freezer
Dec 22 22:12:51 hal lxd.daemon[6196]:   2: fd:   9: perf_event
Dec 22 22:12:51 hal lxd.daemon[6196]:   3: fd:  10: net_cls,net_prio
Dec 22 22:12:51 hal lxd.daemon[6196]:   4: fd:  11: blkio
Dec 22 22:12:51 hal lxd.daemon[6196]:   5: fd:  12: memory
Dec 22 22:12:51 hal lxd.daemon[6196]:   6: fd:  13: pids
Dec 22 22:12:51 hal lxd.daemon[6196]:   7: fd:  14: cpuset
Dec 22 22:12:51 hal lxd.daemon[6196]:   8: fd:  15: cpu,cpuacct
Dec 22 22:12:51 hal lxd.daemon[6196]:   9: fd:  16: hugetlb
Dec 22 22:12:51 hal lxd.daemon[6196]:  10: fd:  17: name=systemd
Dec 22 22:12:51 hal lxd.daemon[6196]: lxcfs.c: 105: do_reload: lxcfs: reloaded
Dec 22 22:12:51 hal lxd.daemon[11357]: => Re-using existing LXCFS
Dec 22 22:12:51 hal lxd.daemon[11357]: => Starting LXD
Dec 22 22:12:51 hal lxd.daemon[11357]: lvl=warn msg="CGroup memory swap accounti
Dec 22 22:12:51 hal lxd.daemon[11357]: lvl=eror msg="Failed to remove device \"/
Dec 22 22:12:51 hal lxd.daemon[11357]: error: remove /var/snap/lxd/common/lxd/de

The output above is column-truncated, any chance you can post a non-truncated version?

Oh, sorry. https://paste.ubuntu.com/26235231/ better?

That’s weird, how can:

Dec 22 22:13:29 hal lxd.daemon[11543]: error: remove /var/snap/lxd/common/lxd/devices/snapcraft-josm/disk.root.: directory not empty

Be a thing if you wiped /var/snap/lxd/common/lxd/devices/*

Followed-up on IRC, a full wipe of /var/snap/lxd/common/lxd/devices followed by a restart of the daemon did the trick to unstick the update.

And the fix has now made it to deb users too (Ubuntu archive, PPA and backports).

1 Like

Many thanks for the help.