LXD not longer working after snap refresh

Hi all,

Out of sudden, my lxd is not logner working. When i use a simple command like ‘lxc list’, it shows an error as below.

user@master-sg:~$ lxc list
cat: /proc/self/attr/current: Permission denied
/snap/lxd/6492/commands/lxc: 6: exec: aa-exec: Permission denied

All my container seems to be still running, but i couldn’t access any of the remote host or containers.

Additional information

user@master-sg:~$ snap changes
ID   Status  Spawn                 Ready                 Summary
40   Done    2018-04-03T10:42:55Z  2018-04-03T10:43:10Z  Auto-refresh snap "lxd"
41   Done    2018-04-03T17:42:55Z  2018-04-03T17:43:09Z  Auto-refresh snap "lxd"
42   Done    2018-04-04T04:17:55Z  2018-04-04T04:18:15Z  Auto-refresh snaps "core", "lxd"
43   Done    2018-04-04T07:06:10Z  2018-04-04T07:06:10Z  Refresh all snaps: no updates
44   Done    2018-04-04T07:19:54Z  2018-04-04T07:19:54Z  Refresh all snaps: no updates

Your help are very much appreciated.

Thank you.

This aa-exec is available in the Ubuntu Core image that snap has.

Do a snap info core and post the result here.
It should say that you have version 16-2.32.1.

Yup correct. What should i do?

name:      core
summary:   snapd runtime environment
publisher: canonical
contact:   snappy-canonical-storeaccount@canonical.com
license:   unknown
description: |
  The core runtime environment for snapd
type:      core
snap-id:   99T7MUlRhtI3U0QFgl5mXXESAiSwt776
tracking:  stable
refreshed: 2018-03-26T19:48:37Z
installed:   16-2.32.1                (4327) 86MB core
channels:                                    
  stable:    16-2.32.1                (4327) 86MB -
  candidate: 16-2.32.1                (4327) 86MB -
  beta:      16-2.32.2                (4376) 90MB -
  edge:      16-2.32.2+git644.abf4628 (4391) 90MB -

We got the same output, so it does not look like a snap core issue.

You can get a shell as the lxd snap, then go and run lxc directly.

snap run --shell lxd
cd /snap/lxd/current/
./command-lxc.wrapper list

With that command, I get my list of the containers. You probably get that error on aa-exec.

Here is the code that invokes aa-exec (in /snap/lxd/current/commands/lxc):

# Re-exec outside of apparmor confinement
if [ -d /sys/kernel/security/apparmor ] && [ "$(cat /proc/self/attr/current)" != "unconfined" ]; then
    exec aa-exec -p unconfined -- "$0" "$@"
fi

Let’s run directly aa-exec, the one from the core image of snap:

bash-4.3$ /snap/core/current/usr/sbin/aa-exec
bash: /snap/core/current/usr/sbin/aa-exec: Permission denied

Weird. I also get Permission denied on the aa-exec of the Ubuntu Core.

Let’s run aa-exec as is. It works for me. Weirdly, it says that it’s in /usr/sbin/aa-exec.

bash-4.3$ aa-exec
USAGE: aa-exec [OPTIONS] <prog> <args>

Confine <prog> with the specified PROFILE.

OPTIONS:
  -p PROFILE, --profile=PROFILE		PROFILE to confine <prog> with
  -n NAMESPACE, --namespace=NAMESPACE	NAMESPACE to confine <prog> in
  -d, --debug				show messages with debugging information
  -i, --immediate			change profile immediately instead of at exec
  -v, --verbose				show messages with stats
  -h, --help				display this help

bash-4.3$ which aa-exec
/usr/sbin/aa-exec

But, earlier, I run (as root, in a different terminal) the command sudo mv /usr/sbin/aa-exec /usr/sbin/aa-exec.stashed-away. What gives?

Let’s compare the two aa-exec files!
Strange, it can still find the aa-exec even if I renamed it. Perhaps it has somewhat opened it or something?
But it cannot find the /usr/sbin/aa-exec.stashed-away.

bash-4.3$ ls -l /usr/sbin/aa-exec
-rwxr-xr-x 1 0 0 22696 Feb 21 13:36 /usr/sbin/aa-exec
bash-4.3$ ls -l /usr/sbin/aa-exec.stashed-away
ls: cannot access '/usr/sbin/aa-exec.stashed-away': No such file or directory
bash-4.3$ 

Okay, so the /usr/sbin/aa-exec we are seeing as a lxd snap user, is not the system’s /usr/sbin/aa-exec. It is likely the one in the core image:

bash-4.3$ ls -l /snap/core/current/usr/sbin/aa-exec
-rwxr-xr-x 1 0 0 22696 Feb 21 13:36 /snap/core/current/usr/sbin/aa-exec
bash-4.3$ 

It looks like there is some chroot-style magic when you do the snap run --shell lxd, which makes /snap/core/current/ the new root directory.
Something is not working properly in your case, hence the issue.

Which distro are you running?

I am running on Ubuntu 16.04.4 LTS (xenial).

Snap has some auto-connect issue in the latest version. Details -> https://forum.snapcraft.io/t/auto-connected-interfaces-missing-on-initial-snap-install/4850/7

If you go to the snapcraft forum (sortof) page and look for the thread titled: Disabling automatic refresh for snap from store . Towards the bottom of the thread there is a suggestion for a line in iptables to block all update attempts by snapd. This is something that I have done. You would need to regress your snap version and also consider at least some modification as to the timing of the updates if you don’t want a repeat of same joys PDQ.
Warning - - - the dev team over at snapcraft is nothing like the group here. Here they listen and seem to be largely working with their users. Over at snapcraft they are quite happy to tell you that its their way or the highway (they have done this to at least a few of the answer seekers that I have noted).