Nested LXD with Debian Bullseye

Yeah, this is quite confusing… I poked at it for a bit but didn’t find an obvious solution.

So far what I’ve noticed:

  • Debian 11 forces the use of cgroup2 in systemd. This causes the container to only get access to the bare minimal cgroup setup, no controllers are available which will most likely confused snapd (that relies on the freezer cgroup). This can be worked around with raw.lxc="lxc.init.cmd=/sbin/init systemd.unified_cgroup_hierarchy=0"
  • For some reason, the mount ordering is incorrect. Snapd is supposed to setup a snap.mount entry which causes /snap to be mounted over itself and then be marked as MS_SHARED|MS_REC. This does happen, but AFTER some of the snaps were already mounted. This over-mounting masks some of the entries and causes the mount errors you’re reporting.
  • Even when those two behave, the snap appears unable to access /run/snapd-snap.socket, getting permission denied. I couldn’t find any apparmor rejection which would explain that one. It could be another mount issue where the snap is somehow getting a different view of /run, or it could be an issue with apparmor inside of the Debian container not generating a profile which properly allows it to work.

I kinda ran out of time to investigate it at this point. It’d be interesting to see if an older Debian release hits the same issue as well as maybe trying the Ubuntu development release to see if that also starts hitting problems, this may help pin point the rest of the issues.