Nested lxd in snap 2.59.1 fails with "execv failed: Permission denied" or "cannot execute snapd tool snap-update-ns: Permission denied"

LXD
1

A first heads-up on a bug I just encountered with nested LXD using snap. After updating snapd to 2.59.1, something causes lxd to no longer work. The error on running containers that were updated is:

execv failed: Permission denied

And after reboot or rebuilding a fresh container it is:

cannot execute snapd tool snap-update-ns: Permission denied
snap-update-ns failed with code 1

The environment uses the included docker profile for the container and some manual workarounds for apparmor. I have not yet checked if these specifics impact the case.

This quick workaround on the container confirmed to fix the situation:

sudo snap revert snapd # go back to 2.58.2

Sorry about the brevity, I will be available to provide more information tomorrow if needed.

1 Like
2

I already got around to checking some basics:

Disabling the apparmor workaround did not help. The workaround used for missing apparmor profiles and runs from rc.local:

apparmor_parser --add /var/lib/snapd/apparmor/profiles/snap*

The docker profile is:

- name: docker
  config:
    linux.kernel_modules: overlay, nf_nat
    security.nesting: true
  devices:
    aadisable:
      path: /sys/module/apparmor/parameters/enabled
      source: /dev/null
      type: disk
    fuse:
      path: /dev/fuse
      type: unix-char
3

I have diagnosed the condition to be caused by the apparmor_parser command in combination with the snapd update.

It is easily reproducible as:

lxc launch ubuntu: c1 -p default -p docker
lxc exec c1 bash # the following are inside the container
snap refresh # required until the image comes with the new snap version
apparmor_parser --add /var/lib/snapd/apparmor/profiles/snap*
lxc

I am not sure yet if that apparmor workaround itself is needed anymore. It was originally related to some further nesting problems and adapted from this source:

https://sleeplessbeastie.eu/2020/07/20/how-to-deal-with-missing-apparmor-profiles-for-microk8s-on-lxd/

I saw at least one like so I assume someone else was also affected by this - I will post any further findings.

1 Like
4

Thanks for posting this as it helped me out. FYI, the snapstore no longer publishes snapd 2.58.3. That means new containers that you want to do nested lxd within will not work until you can install 2.58.3. (The revert command wont work because its a fresh install and there is nothing to revert to.). If you have a machine that does have 2.58.3, you can grab the snap and install it manually.

sudo scp remote:/var/lib/snapd/snaps/snapd_18596.snap /var/lib/snapd/snaps/snapd_18596.snap
sudo snap install --dangerous /var/lib/snapd/snaps/snapd_18596.snap
sudo snap install lxd

note this works for me because I am just launching another container of the same os and same architecture on the same host. I cannot speak to how portable the snapd_.snap is…

5

This great. I made a backup of the working snap using your trick, and now I can also deploy more containers until I get to investigate the full solution or the upstream snapd gets fixed.

6

Turns out these issues were ultimately caused by workaround for previous issues. Removing all workarounds and just using security.nesting=true works.

In other words, using old docker LXD profiles is not helpful, and we also had some apparmor workarounds in place, at least a echo 'tmpfs /sys/kernel/security/ tmpfs defaults 0 0' >> /etc/fstab in the build script that also had to go.

1 Like