LXD install intermittently failing with exec: aa-exec: Permission denied

LXD
#1

Hi,

We’re having trouble installing the LXD snap for snapcraft’s and charmcraft’s github workflows.

This is the error:

+ snap install lxd
error: cannot perform the following tasks:
- Run configure hook of "lxd" snap if present (run hook "configure": 
-----
cat: /proc/self/attr/current: Permission denied
/snap/lxd/23680/snap/hooks/configure: 5: exec: aa-exec: Permission denied
-----)
-----

The environment setup is brief. We apply the following steps to a stock ubuntu image:

  apt-get install -y snapd
  snap install snapd
  snap wait system seed.loaded

  if [ "$SPREAD_SYSTEM" = "ubuntu-18.04-64" ] || [ "$SPREAD_SYSTEM" = "ubuntu-20.04-64" ]; then
      # Remove lxd and lxd-client deb packages as our implementation (pylxd) does not
      # nicely handle the snap and deb being installed at the same time.
      apt-get remove --purge --yes lxd lxd-client
  fi
  # Install and setup the lxd snap
  snap install lxd
  # Add the ubuntu user to the lxd group.
  adduser ubuntu lxd
  lxd init --auto

The biggest mystery for us is the inconsistency. LXD installs fine most of the time. Maybe 10-15% of the time, it fails with this error.

I’m not sure when this started (github logs expire quickly), but it has certainly been occurring more often in the last 2-3 weeks.

Any help or advice? Or is this more of a snapd issue?

#2

This seems related, and may be useful:

#3

Thanks for the link. I tried the 3 primary suggestions in that thread:

  1. Running snap interfaces (now snap connections)
    This was suggested to check if the LXD snap is connected to lxd-support interface.
    This displays nothing, because the LXD snap is not installed yet.

  2. Running snap install core20 and snap revert core20
    This gives the error:

  + snap revert core20

  error: cannot revert "core20": no revision to revert to
  1. Running snap install core20 and snap refresh core20
    This still produces the same exec: aa-exec: Permission denied error.

I’ve captured the journal of a failure, with the failing section here:

oct142128-679423 audit[1763]: AVC apparmor="DENIED" operation="open" profile="snap.lxd.activate" name="/proc/1763/attr/current" pid=1763 comm="cat" requested_mask="r" denied_mask="r" fsuid=0 ouid=0
oct142128-679423 audit[1764]: AVC apparmor="DENIED" operation="exec" profile="snap.lxd.activate" name="/usr/bin/aa-exec" pid=1764 comm="daemon.activate" requested_mask="x" denied_mask="x" fsuid=0 ouid=0
oct142128-679423 audit[1764]: AVC apparmor="DENIED" operation="exec" profile="snap.lxd.activate" name="/usr/bin/aa-exec" pid=1764 comm="daemon.activate" requested_mask="x" denied_mask="x" fsuid=0 ouid=0
oct142128-679423 kernel: kauditd_printk_skb: 4 callbacks suppressed
oct142128-679423 kernel: audit: type=1400 audit(1665782994.672:46): apparmor="DENIED" operation="open" profile="snap.lxd.activate" name="/proc/1763/attr/current" pid=1763 comm="cat" requested_mask="r" denied_mask="r" fsuid=0 ouid=0
oct142128-679423 kernel: audit: type=1400 audit(1665782994.672:47): apparmor="DENIED" operation="exec" profile="snap.lxd.activate" name="/usr/bin/aa-exec" pid=1764 comm="daemon.activate" requested_mask="x" denied_mask="x" fsuid=0 ouid=0
oct142128-679423 kernel: audit: type=1400 audit(1665782994.672:48): apparmor="DENIED" operation="exec" profile="snap.lxd.activate" name="/usr/bin/aa-exec" pid=1764 comm="daemon.activate" requested_mask="x" denied_mask="x" fsuid=0 ouid=0
oct142128-679423 lxd.activate[1744]: The LXD snap was unable to run aa-exec, this usually indicates a LXD sideload.
oct142128-679423 lxd.activate[1744]: When sideloading, make sure to manually connect all interfaces.
oct142128-679423 systemd[1]: snap.lxd.activate.service: Deactivated successfully.

Any other ideas or advice?

#4

I’m not an expert in snap internals or operations.

But are you trying to install LXD snap inside another LXD container by any chance?

As I’m aware there are issues with AppArmor and nested containers (although I would have expected that to hit every time not intermittently).

#5

It might be worth posting over on the snapd forums too and see if anyone there knows what is going on with it.

#6

Do you have a set of reproducer steps I can try using just LXD commands?

#7

But are you trying to install LXD snap inside another LXD container by any chance?

The tests are run via spread in the Google Compute Platform, which I believe are just VMs with a stock Ubuntu image.

It might be worth posting over on the snapd forums too and see if anyone there knows what is going on with it.

Good idea. I’ll cross-post there (link). I’m leaning more towards this being a snapd issue.

Do you have a set of reproducer steps I can try using just LXD commands?

I don’t have steps to reproduce reliably, it’s very rare. The steps are

  1. Spawn a new VM (ubuntu 18.04, 20.04, or 22.04)
  2. Run these commands:
apt-get install -y snapd
snap install snapd
snap wait system seed.loaded
snap install lxd

And the failure rate for these steps is very low - less than 1% of the time. In our github workflow, we run ~350 spread tests, so the failure of the overall work flow is around 10-15% (although some weeks it has been closer to 50%).

1 Like
#8

@stgraber have you seen anything like this before?

#9

That would be a snapd bug. The failure suggests that the lxd-support interface isn’t connected.

1 Like
#10

Thanks!

I’m asking the snapd team on the snapcraft forum and in launchpad.

1 Like