Security.privileged true needed with shift=true on some systems/containers

I have an ubuntu 20.04 host running lxd installed via snap and using a zfs backend, & after an apt update & apt upgrade (which i do regularly as needed), kernel is 5.15.0 (not sure what it was prior to the apt upgrade). Lxd is now version 5.5-37534be as reported by snap

That update seems to have borked lxd in a way i’ve never seen before (been running lxd since 2.x days). One of my containers would start but i couldn’t connect to it’s web UI and 2 containers would not start at all.

the 2 containers that wouldn’t start both have disk devices added with shift=true.

The container that would start but that i couldn’t connect to the web UI gave an OSError:
"OSError: [Errno 75] Value too large for defined data type..."
unsure if this was associated with lxd, snap, or my application (but i didn’t find any related app issue upon googling).

long story short, i exported the containers and moved them to another host - where they work just fine… the other system is also ubuntu 20.04, but running kernel 5.4 (& same version of lxd as above). Also ran apt update & apt upgrade, but didn’t suffer any of these other issues.

So, on the 5.15 kernel system, i deleted all the containers, removed and reinstalled LXD & then I re-imported the containers and the one giving the OSError, now works fine.

However, the 2 containers that have disk devices added with shiftfs enabled dont’ start. They both give the same error (just different container name) - here’s one:

Error: Failed to run: /snap/lxd/current/bin/lxd forkstart owntone /var/snap/lxd/common/lxd/containers /var/snap/lxd/common/lxd/logs/owntone/lxc.conf: exit status 1
Try `lxc info --show-log owntone` for more info..

the result of lxc info --show-log owntone:

Name: owntone
Status: STOPPED
Type: container
Architecture: x86_64
Created: 2022/09/21 22:55 MDT
Last Used: 2022/09/22 14:34 MDT

Log:

lxc owntone 20220922203457.187 WARN     conf - ../src/src/lxc/conf.c:lxc_map_ids:3592 - newuidmap binary is missing
lxc owntone 20220922203457.187 WARN     conf - ../src/src/lxc/conf.c:lxc_map_ids:3598 - newgidmap binary is missing
lxc owntone 20220922203457.188 WARN     conf - ../src/src/lxc/conf.c:lxc_map_ids:3592 - newuidmap binary is missing
lxc owntone 20220922203457.188 WARN     conf - ../src/src/lxc/conf.c:lxc_map_ids:3598 - newgidmap binary is missing
lxc owntone 20220922203457.188 WARN     cgfsng - ../src/src/lxc/cgroups/cgfsng.c:fchowmodat:1611 - No such file or directory - Failed to fchownat(42, memory.oom.group, 1000000000, 0, AT_EMPTY_PATH | AT_SYMLINK_NOFOLLOW )
lxc owntone 20220922203457.352 ERROR    conf - ../src/src/lxc/conf.c:run_buffer:321 - Script exited with status 1
lxc owntone 20220922203457.352 ERROR    conf - ../src/src/lxc/conf.c:lxc_setup:4400 - Failed to run mount hooks
lxc owntone 20220922203457.352 ERROR    start - ../src/src/lxc/start.c:do_start:1272 - Failed to setup container "owntone"
lxc owntone 20220922203457.352 ERROR    sync - ../src/src/lxc/sync.c:sync_wait:34 - An error occurred in another process (expected sequence number 4)
lxc owntone 20220922203457.359 WARN     network - ../src/src/lxc/network.c:lxc_delete_network_priv:3631 - Failed to rename interface with index 0 from "eth0" to its initial name "veth7501d124"
lxc owntone 20220922203457.360 ERROR    lxccontainer - ../src/src/lxc/lxccontainer.c:wait_on_daemonized_start:877 - Received container state "ABORTING" instead of "RUNNING"
lxc owntone 20220922203457.360 ERROR    start - ../src/src/lxc/start.c:__lxc_start:2107 - Failed to spawn container "owntone"
lxc owntone 20220922203457.360 WARN     start - ../src/src/lxc/start.c:lxc_abort:1036 - No such process - Failed to send SIGKILL via pidfd 43 for process 782016
lxc owntone 20220922203457.460 WARN     conf - ../src/src/lxc/conf.c:lxc_map_ids:3592 - newuidmap binary is missing
lxc owntone 20220922203457.460 WARN     conf - ../src/src/lxc/conf.c:lxc_map_ids:3598 - newgidmap binary is missing
lxc 20220922203457.498 ERROR    af_unix - ../src/src/lxc/af_unix.c:lxc_abstract_unix_recv_fds_iov:218 - Connection reset by peer - Failed to receive response
lxc 20220922203457.499 ERROR    commands - ../src/src/lxc/commands.c:lxc_cmd_rsp_recv_fds:128 - Failed to receive file descriptors for command "get_state"

However, if I lxc config set owntone security.privileged true, then the container starts & runs fine.

I didn’t need to set security.privileged to true on the kernel 5.4 system… (& prior to the update of the kernel 5.15 system, i also didn’t have those containers set to use privileged mode)

One thing i noticed as different when i ran lxc info on both systems is the following kernel feature:

idmapped_mounts: “true” on the kernel 5.15 system, whereas
idmapped_mounts: “false” on the kernel 5.4 system

I noted this here because i saw another post that warns that “shiftfs shouldn’t be mounted on top of idmapped mounts …”, but (a) wasn’t sure if this is what is happening and (b) i can’t figure out how to set idmapped_mounts to false assuming that is possible and would help…

appreciate any help - i’d like to get the containers running without having to rely on privileged mode

i believe the issue resulted from enabling shiftfs - which isn’t needed any longer on ubuntu kernels > 5.12 if using a supported file system - on a kernel 5.15 system which has idmapped_mounts enabled.

I unset the shiftfs key in the snap, reloaded the lxd snap, unset security.privileged in the containers and then removed and readded (without the key shift=true) the folder to be shared. Now it all works as it should, except the shared folder shows up with owner:group of nobody:nogroup. So, I added a raw.idmap to the container and the mapping works now as expected.

The folder i’m sharing originates as a cifs shared folder on TrueNAS (and the underlying file system there is zfs), which is mounted as a shared folder on my lxd host. In turn, i then shared it with the container… I’m presuming the underlying zfs file system (currently not supported by idmapped_mounts) is why i needed to add the raw.idmaps and that idmapped_mounts didn’t automatically just work.

Is there a way to use shiftfs still on kernels that support idmapped_mounts and not require a privileged container?

Yes I’ve been experiencing the same thing as you and came to the same conclusions, see:

It seems like a recent kernel regression.

You are correct that idmapped mounts replaces shiftfs functionality, but currently doesn’t work on ZFS or TMPFS filesystems.

We’ve opened a bug to report the kernel regression:

appreciate the followup, thanks