Can't enable lxd snap any longer

debian 9
lxd 3.15 (running from snap package)

as containers can not connect to the internet by default I regularly run sudo snap disable lxd && sudo snap enable lxd after a system reboot. This makes internet accessible to my containers

just today I can re-enable lxd any longer

sudo snap enable lxd
error: cannot perform the following tasks:
- Start snap "lxd" (11381) services ([start snap.lxd.activate.service] failed with exit status 1: Job for snap.lxd.activate.service failed because the control process exited with error code.
See "systemctl status snap.lxd.activate.service" and "journalctl -xe" for details.
)

the logs I am advised to look at do not tell me much though


systemctl status snap.lxd.activate.service

● snap.lxd.activate.service
   Loaded: not-found (Reason: No such file or directory)
   Active: failed (Result: exit-code) since Tue 2019-07-30 11:19:05 CEST; 8min ago
 Main PID: 5224 (code=exited, status=1/FAILURE)
      CPU: 96ms

sudo journalctl -xe

-- Defined-By: systemd
-- Support: https://www.debian.org/support
-- 
-- Unit snap.lxd.activate.service has failed.
-- 
-- The result is failed.
Jul 30 11:19:05 debian systemd[1]: snap.lxd.activate.service: Unit entered failed state.
Jul 30 11:19:05 debian systemd[1]: snap.lxd.activate.service: Failed with result 'exit-code'.
Jul 30 11:19:07 debian systemd[1]: Reloading.
Jul 30 11:19:07 debian systemd[1]: apt-daily-upgrade.timer: Adding 23min 6.676495s random time.
Jul 30 11:19:07 debian systemd[1]: anacron.timer: Adding 4min 35.435614s random time.
Jul 30 11:19:07 debian systemd[1]: apt-daily.timer: Adding 2h 59min 16.913487s random time.
Jul 30 11:19:07 debian systemd[1]: snapd.refresh.timer: Adding 4min 30.586035s random time.
Jul 30 11:19:07 debian systemd[1]: Starting Automatically refresh installed snaps...
-- Subject: Unit snapd.refresh.service has begun start-up
-- Defined-By: systemd
-- Support: https://www.debian.org/support
-- 
-- Unit snapd.refresh.service has begun starting up.
Jul 30 11:19:07 debian /usr/bin/snap[6003]: cmd.go:105: DEBUG: restarting into "/snap/core/current/usr/bin/snap"
Jul 30 11:19:07 debian snap[6003]: Ignoring `snap refresh` from the systemd timer
Jul 30 11:19:07 debian systemd[1]: Started Automatically refresh installed snaps.
-- Subject: Unit snapd.refresh.service has finished start-up
-- Defined-By: systemd
-- Support: https://www.debian.org/support
-- 
-- Unit snapd.refresh.service has finished starting up.
-- 
-- The start-up result is done.
Jul 30 11:19:07 debian systemd[1]: snapd.refresh.timer: Adding 4h 47min 13.593423s random time.
Jul 30 11:19:07 debian systemd[1]: snapd.refresh.timer: Adding 4h 38min 32.447457s random time.
Jul 30 11:19:09 debian snapd[1176]: handlers.go:437: Reported install problem for "lxd" as already-reported
Jul 30 11:19:10 debian sudo[5183]: pam_unix(sudo:session): session closed for user root
Jul 30 11:21:00 debian gnome-keyring-daemon[2728]: asked to register item /org/freedesktop/secrets/collection/login/43, but it's already registered
Jul 30 11:21:29 debian gnome-keyring-daemon[2728]: asked to register item /org/freedesktop/secrets/collection/login/43, but it's already registered
Jul 30 11:23:55 debian snap[6342]: cmd.go:105: DEBUG: restarting into "/snap/core/current/usr/bin/snap"
Jul 30 11:26:11 debian sudo[6387]:   [myuser] : TTY=pts/2 ; PWD=/home/[myuser] ; USER=root ; COMMAND=/bin/journalctl -xe
Jul 30 11:26:11 debian sudo[6387]: pam_unix(sudo:session): session opened for user root by (uid=0)

can anyone help with this?

Can you show find /etc/systemd | grep snap?

One trick that works sometimes if the issue is missing files on disk is:

  • snap refresh lxd --candidate
  • snap refresh lxd --stable

This effectively forces snapd to re-setup all the units, confinement, …

This may sort out your problem.

you can’t apparently refresh a disabled snap, so there is no cure in this unfortunately

find /etc/systemd/ | grep snap
/etc/systemd/system/multi-user.target.wants/snapd.autoimport.service
/etc/systemd/system/multi-user.target.wants/snapd.service
/etc/systemd/system/multi-user.target.wants/snap-electronic\x2dwechat-7.mount
/etc/systemd/system/multi-user.target.wants/snap-chromium\x2dffmpeg-9.mount
/etc/systemd/system/multi-user.target.wants/snap-gtk\x2dcommon\x2dthemes-1198.mount
/etc/systemd/system/multi-user.target.wants/snap-chromium\x2dffmpeg-11.mount
/etc/systemd/system/multi-user.target.wants/snap-chromium\x2dffmpeg-13.mount
/etc/systemd/system/multi-user.target.wants/snap-gimp-165.mount
/etc/systemd/system/multi-user.target.wants/snap-gtk2\x2dcommon\x2dthemes-4.mount
/etc/systemd/system/multi-user.target.wants/snap-obs\x2dstudio-519.mount
/etc/systemd/system/multi-user.target.wants/snap-gimp-189.mount
/etc/systemd/system/multi-user.target.wants/snap-shotcut-47.mount
/etc/systemd/system/multi-user.target.wants/snap-obs\x2dstudio-525.mount
/etc/systemd/system/multi-user.target.wants/snap-core-7169.mount
/etc/systemd/system/multi-user.target.wants/snap-telegram\x2ddesktop-818.mount
/etc/systemd/system/multi-user.target.wants/snap-core-7270.mount
/etc/systemd/system/multi-user.target.wants/snap-audacity-377.mount
/etc/systemd/system/multi-user.target.wants/snap-gtk2\x2dcommon\x2dthemes-5.mount
/etc/systemd/system/multi-user.target.wants/snap-core18-1055.mount
/etc/systemd/system/multi-user.target.wants/snap-gtk\x2dcommon\x2dthemes-1313.mount
/etc/systemd/system/multi-user.target.wants/snap-telegram\x2ddesktop-836.mount
/etc/systemd/system/multi-user.target.wants/snap-opera-43.mount
/etc/systemd/system/multi-user.target.wants/snap-core18-1066.mount
/etc/systemd/system/multi-user.target.wants/snap-audacity-408.mount
/etc/systemd/system/multi-user.target.wants/snap-shotcut-49.mount
/etc/systemd/system/multi-user.target.wants/snap-lxd-11353.mount
/etc/systemd/system/multi-user.target.wants/snap-opera-44.mount
/etc/systemd/system/multi-user.target.wants/snap-atom-234.mount
/etc/systemd/system/multi-user.target.wants/snap-lxd-11381.mount
/etc/systemd/system/multi-user.target.wants/snap-atom-235.mount
/etc/systemd/system/sockets.target.wants/snapd.socket
/etc/systemd/system/timers.target.wants/snapper-boot.timer
/etc/systemd/system/timers.target.wants/snapper-cleanup.timer
/etc/systemd/system/timers.target.wants/snapper-timeline.timer
/etc/systemd/system/timers.target.wants/snapd.refresh.timer
/etc/systemd/system/snap-electronic\x2dwechat-7.mount
/etc/systemd/system/snap-chromium\x2dffmpeg-9.mount
/etc/systemd/system/snap-gtk\x2dcommon\x2dthemes-1198.mount
/etc/systemd/system/snap-chromium\x2dffmpeg-11.mount
/etc/systemd/system/snap-chromium\x2dffmpeg-13.mount
/etc/systemd/system/snap-gimp-165.mount
/etc/systemd/system/snap-gtk2\x2dcommon\x2dthemes-4.mount
/etc/systemd/system/snap-obs\x2dstudio-519.mount
/etc/systemd/system/snap-gimp-189.mount
/etc/systemd/system/snap-shotcut-47.mount
/etc/systemd/system/snap-obs\x2dstudio-525.mount
/etc/systemd/system/snap-core-7169.mount
/etc/systemd/system/snap-telegram\x2ddesktop-818.mount
/etc/systemd/system/snap-core-7270.mount
/etc/systemd/system/snap-audacity-377.mount
/etc/systemd/system/snap-gtk2\x2dcommon\x2dthemes-5.mount
/etc/systemd/system/snap-core18-1055.mount
/etc/systemd/system/snap-gtk\x2dcommon\x2dthemes-1313.mount
/etc/systemd/system/snap-telegram\x2ddesktop-836.mount
/etc/systemd/system/snap-opera-43.mount
/etc/systemd/system/snap-core18-1066.mount
/etc/systemd/system/snap-audacity-408.mount
/etc/systemd/system/snap-shotcut-49.mount
/etc/systemd/system/snap-lxd-11353.mount
/etc/systemd/system/snap-opera-44.mount
/etc/systemd/system/snap-atom-234.mount
/etc/systemd/system/snap-lxd-11381.mount
/etc/systemd/system/snap-atom-235.mount

Can you show the output of snap changes?

snap changes 
ID   Status  Spawn                Ready                Summary
309  Done    today at 08:17 CEST  today at 08:18 CEST  Disable "lxd" snap
310  Done    today at 08:18 CEST  today at 08:19 CEST  Enable "lxd" snap
311  Done    today at 08:26 CEST  today at 08:26 CEST  Disable "lxd" snap
312  Error   today at 08:27 CEST  today at 08:27 CEST  Enable "lxd" snap
313  Error   today at 08:28 CEST  today at 08:28 CEST  Enable "lxd" snap
314  Error   today at 11:12 CEST  today at 11:13 CEST  Enable "lxd" snap
315  Error   today at 11:18 CEST  today at 11:19 CEST  Enable "lxd" snap
316  Error   today at 12:56 CEST  today at 12:56 CEST  Enable "lxd" snap
317  Error   today at 13:15 CEST  today at 13:16 CEST  Enable "lxd" snap

that seems just to confirm the status quo (lxd snap can’t be enabled)

Hmm, okay, well, that’s quite frustrating.

One way around this that I can think of would be to just manually copy those units from another system, they’re normally located in /etc/systemd/system.

Once copied, run systemctl daemon-reload, then try the snap enable lxd again.

I would also really appreciate it if you could report this at https://forum.snapcraft.io as this is clearly not a normal behavior and should absolutely get looked at.

Can you also show snap change 317, this may show more details too.

in the meantime and several restarts later lxc is enables again but does not work

lxc list
Error: Get http://unix.socket/1.0: dial unix /var/snap/lxd/common/lxd/unix.socket: connect: resource temporarily unavailable

trying to follow the recommended snap refresh lxc --candidate freezes at:

Start snap "lxd" (11437) services

on the problematic system I get

ls -l /etc/systemd/system | grep lxd
-rw-r--r-- 1 root root  227 Jul 25 14:26 snap-lxd-11353.mount
-rw-r--r-- 1 root root  227 Jul 26 21:12 snap-lxd-11381.mount
-rw-r--r-- 1 root root  227 Aug  1 06:01 snap-lxd-11437.mount
-rw-r--r-- 1 root root  396 Aug  1 06:02 snap.lxd.activate.service
-rw-r--r-- 1 root root  470 Aug  1 06:02 snap.lxd.daemon.service
-rw-r--r-- 1 root root  330 Aug  1 06:02 snap.lxd.daemon.unix.socket

can someone specify which files as per stephanes comment above are missing here? As I do not have a second system running lxd from the snap package I have nothing to compare this to

snap change 317
error: cannot find change with id "317"

would it be a way to remove the lxd snap an re-install it? I guess it won’t hurt to take a backup of
/var/snap/lxd/common/lxd/storage-pools/* (where the actual data lives) beforehand?

Compared to a working install I don’t see something wrong except for the 11437 - but it’s what @stgraber asked you to do - and the missing 11405 but from the snap forum messages you have now refreshed to 11405 successfully. So I’m left with the feeling that something could be wrong on the lxd side. Did you look at
sudo cat /var/snap/lxd/common/lxd/logs/lxd.log
to see if there anything interesting in it ?

t=2019-08-01T16:46:42+0200 lvl=info msg="LXD 3.15 is starting in normal mode" path=/var/snap/lxd/common/lxd
t=2019-08-01T16:46:42+0200 lvl=info msg="Kernel uid/gid map:" 
t=2019-08-01T16:46:42+0200 lvl=info msg=" - u 0 0 4294967295" 
t=2019-08-01T16:46:42+0200 lvl=info msg=" - g 0 0 4294967295" 
t=2019-08-01T16:46:42+0200 lvl=info msg="Configured LXD uid/gid map:" 
t=2019-08-01T16:46:42+0200 lvl=info msg=" - u 0 1000000 1000000000" 
t=2019-08-01T16:46:42+0200 lvl=info msg=" - g 0 1000000 1000000000" 
t=2019-08-01T16:46:42+0200 lvl=warn msg="AppArmor support has been disabled because of lack of kernel support" 
t=2019-08-01T16:46:42+0200 lvl=warn msg="CGroup memory swap accounting is disabled, swap limits will be ignored." 
t=2019-08-01T16:46:42+0200 lvl=info msg="Kernel features:" 
t=2019-08-01T16:46:42+0200 lvl=info msg=" - netnsid-based network retrieval: no" 
t=2019-08-01T16:46:42+0200 lvl=info msg=" - uevent injection: no" 
t=2019-08-01T16:46:42+0200 lvl=info msg=" - seccomp listener: no" 
t=2019-08-01T16:46:42+0200 lvl=info msg=" - unprivileged file capabilities: no" 
t=2019-08-01T16:46:42+0200 lvl=info msg=" - shiftfs support: no" 
t=2019-08-01T16:46:42+0200 lvl=info msg="Initializing local database" 
t=2019-08-01T16:46:42+0200 lvl=eror msg="Failed to start the daemon: Failed to start dqlite server: run failed with 13" 
t=2019-08-01T16:46:42+0200 lvl=info msg="Starting shutdown sequence" 

I am terrible with logs. The only thing that seems to pop out here is
"Failed to start the daemon: Failed to start dqlite server: run failed with 13"

which I can find in another Topic here on the forum from a while ago (also occuring with 3.15), so I’ll try to see whether I can find some cure in that thread.