Changed zfs to dir, preseed init succeeded, and successfully launched nested container. Thank you!
Next is an error with snapd and apparmor.
Using a very basic yaml:
config:
core.https_address: '[::]:8443'
core.trust_password: ""
networks:
- config:
ipv4.address: auto
ipv6.address: auto
description: ""
name: lxdbr0
type: ""
storage_pools:
- config: {}
description: ""
name: default
driver: dir
profiles:
- config: {}
description: ""
devices:
eth0:
name: eth0
network: lxdbr0
type: nic
root:
path: /
pool: default
type: disk
name: default
cluster: null
The layout so far:
- host001 (test VM host on physical host) is now running the following:
- org001 (main container), which is now running the following:
- web001 (nested container)
Created local image with LXD preinstalled to avoid downloading new image every time.
Overview of steps so far:
- VM host: update, upgrade, install snapd, snap install lxd (4.0/stable)
- Update /etc/subgid, /etc/subuid, then restart VM
- Reconnect: retrieve yaml, lxd init, launch org001 (security nesting true)
- Connect to org001, update with new subgid & subuid, snap restart lxd, unable to use snap
error: system does not fully support snapd: apparmor detected but insufficient permissions to use it
Would like to launch another nested LXD container within this one.
Looked for possible cause of insufficient permissions.
From root@web001
:
# which snap
/usr/bin/snap
# apparmor_status | grep snap
You do not have enough privilege to read the profile set.
Found /etc/apparmor.d/usr.lib.snapd.snap-confine.real
, went through it, it seemed okay, everything default, although it would seem weird to need to manually edit that.
Maybe it needed to be reinstalled directly within web001.
# apt-get install --reinstall snapd
Attempted to restard LXD.
# snap restart lxd
Same error, apparmor detected insufficient permissions to use snapd.
Installed apparmor-utils, still unable to restart lxd, reinstalled snapd again, new error.
# apt-get install apparmor-utils
# snap restart lxd
Same error, apparmor detected insufficient permissions to use snapd.
# apt-get install --reinstall snapd
apparmor_parser: Unable to replace "mount-namespace-capture-helper". Permission denied; attempted to load a profile while confined? apparmor_parser: Unable to replace "/usr/lib/snapd/snap-confine". Permission denied; attempted to load a profile while confined? snapd.failure.service is a disabled or a static unit, not starting it. snapd.snap-repair.service is a disabled or a static unit, not starting it. Job for snapd.apparmor.service failed because the control process exited with error code. See "systemctl status snapd.apparmor.service" and "journalctl -xe" for details. Processing triggers for mime-support (3.64ubuntu1) ...
# systemctl status snapd.apparmor.service
● snapd.apparmor.service - Load AppArmor profiles managed internally by snapd
Loaded: loaded (/lib/systemd/system/snapd.apparmor.service; enabled; vendor preset: enabled)
Active: failed (Result: exit-code) since Tue 2020-06-23 18:05:26 UTC; 7min ago
Process: 994 ExecStart=/usr/lib/snapd/snapd-apparmor start (code=exited, status=123)
Main PID: 994 (code=exited, status=123)
Error could not replace profile snap.lxd.lxc-to-lxd (and many more), permission denied.
It asked if attempted to load profile while confined.
Maybe there is a need for a configuration change in apparmor.
snapd.apparmor.service: Main process exited, code=exited, status=123/n/a
web001 systemd[1]: snapd.apparmor.service: Failed with result 'exit-code'.
web001 systemd[1]: Failed to start Load AppArmor profiles managed internally by snapd.
# journalctl -xe
highlights:
apparmor.systemd[109]: Error: Could not replace profile /var/cache/apparmor/26b63962.0/lsb_release: Permission denied
snapd-apparmor[158]: Error: Could not replace profile /var/cache/apparmor/26b63962.0/snap-update-ns.lxd: Permission denied
…
I might delete the web001 container, create a new one and try fresh without the reinstalls, or maybe try a different image if the error is due to an image created within another container, although would like to be able to use one image as a general LXD template, there will be 2 more layers of containers after web001, including 3 containers within it (for 3 environments), and each of those will contain the actual applications.
Update:
This error:
error: system does not fully support snapd: apparmor detected but insufficient permissions to use it
Also occurs with normal images. Tried spinning up from images:ubuntu/focal.
Some googling may find something, just from the error output alone it looks like it is a setting in apparmor disallowing snap/snapd from doing its thing.
On second thought, maybe it’s what it says about the system not supporting it. Maybe the parent container (or host beyond that) needs updated to allow child/nested containers to have more permissions.
From a quick search this may resolve it:
# lxc profile edit default
Add the following:
config:
security.nesting: "true"
security.privileged: "true"
Possibly also:
raw.lxc: |-
lxc.cgroup.devices.allow=a
lxc.mount.auto=proc:rw sys:rw
Going to give that a try.
Okay, so far it may only need the config:
update in profile, have not tried the raw.lxc
part yet., although snap refresh
works now, still unable to restart lxd.
# snap refresh
All snaps up to date.
# snap restart lxd
error: cannot perform the following tasks:
- restart of [lxd.activate lxd.daemon] (# systemctl restart snap.lxd.activate.service snap.lxd.daemon.service
Job for snap.lxd.activate.service failed because the control process exited with error code.
See "systemctl status snap.lxd.activate.service" and "journalctl -xe" for details.
)
- restart of [lxd.activate lxd.daemon] (exit status 1)
# systemctl status snap.lxd.activate.service
● snap.lxd.activate.service - Service for snap application lxd.activate
Loaded: loaded (/etc/systemd/system/snap.lxd.activate.service; enabled; vendor preset: enabled)
Active: failed (Result: exit-code) since Tue 2020-06-23 21:56:39 UTC; 36s ago
Process: 306 ExecStart=/usr/bin/snap run lxd.activate (code=exited, status=1/FAILURE)
Main PID: 306 (code=exited, status=1/FAILURE)
systemd[1]: Starting Service for snap application lxd.activate...
lxd.activate[339]: cannot change apparmor hat: No child processes
lxd.activate[340]: cannot change profile for the next exec call: Permission denied
systemd[1]: snap.lxd.activate.service: Main process exited, code=exited, status=1/FAILURE
lxd.activate[306]: snap-update-ns failed with code 1: File exists
systemd[1]: snap.lxd.activate.service: Failed with result 'exit-code'.
systemd[1]: Failed to start Service for snap application lxd.activate.
journalctl
systemd-logind.service: Unexpected error response from GetNameOwner(): Connection terminated
systemd[1]: systemd-logind.service: start operation timed out. Terminating.
systemd[1]: systemd-logind.service: Unexpected error response from GetNameOwner(): Connection terminated
systemd[1]: systemd-logind.service: Failed with result 'timeout'.
Maybe the raw.lxc
change will resolve it.
Update 2:
Actually, I had to reboot the parent container.
# systemctl status snap.lxd.activate.service
● snap.lxd.activate.service - Service for snap application lxd.activate
Loaded: loaded (/etc/systemd/system/snap.lxd.activate.service; enabled; vendor preset: enabled)
Active: activating (start) since Tue 2020-06-23 22:04:10 UTC; 326ms ago
Main PID: 184 (snap)
Tasks: 1 (limit: 14276)
Memory: 7.4M
CGroup: /system.slice/snap.lxd.activate.service
└─184 /usr/bin/snap run lxd.activate
systemd[1]: Starting Service for snap application lxd.activate...
The weird thing is, I used the security nesting true option when creating the containers, didn’t think that would be needed in the profile. Likewise, don’t know if it’s necessary to set security privileged true at the profile level. Going to try a fresh parent container, reboot it, then spin up a child container, see if the permissions go through. If not, then may just need to go ahead with a revised profile.
I would like to avoid setting security.privileged true, I have already created new ranges for uid/gid, would prefer keeping container IDs separate from host. I’m pretty sure this worked in the past without having to set that value to true, although the security.nesting has required true.
Update 3:
Tried various things, at first I thought it might require security.privileged: "true"
in profile (unless somewhere else is better), although I would rather not use that. Maybe it’s an issue with /etc/subgid
and /etc/subuid
, although I have a larger range for each parent-level container, with the most on the host.
With security privileged true, I am able to successfully run snap refresh
and snap refresh lxd
, although when I run snap restart lxd
it fails:
error: cannot perform the following tasks:
- restart of [lxd.activate lxd.daemon] (# systemctl restart snap.lxd.activate.service snap.lxd.daemon.service
Job for snap.lxd.activate.service failed because the control process exited with error code.
See "systemctl status snap.lxd.activate.service" and "journalctl -xe" for details.
)
- restart of [lxd.activate lxd.daemon] (exit status 1)
Logs show same output as before.
Status shows not running. I restarted the container before checking.
# systemctl status snap.lxd.activate.service
● snap.lxd.activate.service - Service for snap application lxd.activate
Loaded: loaded (/etc/systemd/system/snap.lxd.activate.service; enabled; vendor preset: enabled)
Active: failed (Result: exit-code) since Wed 2020-06-24 01:02:14 UTC; 8s ago
Process: 172 ExecStart=/usr/bin/snap run lxd.activate (code=exited, status=1/FAILURE)
Main PID: 172 (code=exited, status=1/FAILURE)
systemd[1]: Starting Service for snap application lxd.activate...
lxd.activate[210]: cannot change apparmor hat: No child processes
lxd.activate[213]: cannot change profile for the next exec call: Permission denied
lxd.activate[172]: snap-update-ns failed with code 1: No such file or directory
systemd[1]: snap.lxd.activate.service: Main process exited, code=exited, status=1/FAILURE
systemd[1]: snap.lxd.activate.service: Failed with result 'exit-code'.
systemd[1]: Failed to start Service for snap application lxd.activate.
To avoid going in circles, going to step away, do some extra digging into this later.
It’s probably a really simple fix to permissions and I’m not seeing it.
For some additional context, here are the subgid & subuid numbers I’m using:
host001 (vm with base lxd):
echo "root:2345678901:2345678901" | tee -a /etc/subuid /etc/subgid && echo "lxd:1234567890:1234567890" | tee -a /etc/subuid /etc/subgid
org001 (lxd within host001):
echo "root:234567890:234567890" | tee -a /etc/subuid /etc/subgid && echo "lxd:123456789:123456789" | tee -a /etc/subuid /etc/subgid
web001 (lxd within org001), this would be set up with the following but currently unable to use lxd in that node:
echo "root:23456789:23456789" | tee -a /etc/subuid /etc/subgid && echo "lxd:12345678:12345678" | tee -a /etc/subuid /etc/subgid
Each time I update the subgid & subuid, I snap restart lxd
.
Maybe my numbers are off.
I would like to add one more…
dev001 (lxd within web001):
echo "root:2345678:2345678" | tee -a /etc/subuid /etc/subgid && echo "lxd:1234567:1234567" | tee -a /etc/subuid /etc/subgid