Security.privileged true needed with shift=true on some systems/containers

LXD
,
1

I have an ubuntu 20.04 host running lxd installed via snap and using a zfs backend, & after an apt update & apt upgrade (which i do regularly as needed), kernel is 5.15.0 (not sure what it was prior to the apt upgrade). Lxd is now version 5.5-37534be as reported by snap

That update seems to have borked lxd in a way i’ve never seen before (been running lxd since 2.x days). One of my containers would start but i couldn’t connect to it’s web UI and 2 containers would not start at all.

the 2 containers that wouldn’t start both have disk devices added with shift=true.

The container that would start but that i couldn’t connect to the web UI gave an OSError:
"OSError: [Errno 75] Value too large for defined data type..."
unsure if this was associated with lxd, snap, or my application (but i didn’t find any related app issue upon googling).

long story short, i exported the containers and moved them to another host - where they work just fine… the other system is also ubuntu 20.04, but running kernel 5.4 (& same version of lxd as above). Also ran apt update & apt upgrade, but didn’t suffer any of these other issues.

So, on the 5.15 kernel system, i deleted all the containers, removed and reinstalled LXD & then I re-imported the containers and the one giving the OSError, now works fine.

However, the 2 containers that have disk devices added with shiftfs enabled dont’ start. They both give the same error (just different container name) - here’s one:

Error: Failed to run: /snap/lxd/current/bin/lxd forkstart owntone /var/snap/lxd/common/lxd/containers /var/snap/lxd/common/lxd/logs/owntone/lxc.conf: exit status 1
Try `lxc info --show-log owntone` for more info..

the result of lxc info --show-log owntone:

Name: owntone
Status: STOPPED
Type: container
Architecture: x86_64
Created: 2022/09/21 22:55 MDT
Last Used: 2022/09/22 14:34 MDT

Log:

lxc owntone 20220922203457.187 WARN     conf - ../src/src/lxc/conf.c:lxc_map_ids:3592 - newuidmap binary is missing
lxc owntone 20220922203457.187 WARN     conf - ../src/src/lxc/conf.c:lxc_map_ids:3598 - newgidmap binary is missing
lxc owntone 20220922203457.188 WARN     conf - ../src/src/lxc/conf.c:lxc_map_ids:3592 - newuidmap binary is missing
lxc owntone 20220922203457.188 WARN     conf - ../src/src/lxc/conf.c:lxc_map_ids:3598 - newgidmap binary is missing
lxc owntone 20220922203457.188 WARN     cgfsng - ../src/src/lxc/cgroups/cgfsng.c:fchowmodat:1611 - No such file or directory - Failed to fchownat(42, memory.oom.group, 1000000000, 0, AT_EMPTY_PATH | AT_SYMLINK_NOFOLLOW )
lxc owntone 20220922203457.352 ERROR    conf - ../src/src/lxc/conf.c:run_buffer:321 - Script exited with status 1
lxc owntone 20220922203457.352 ERROR    conf - ../src/src/lxc/conf.c:lxc_setup:4400 - Failed to run mount hooks
lxc owntone 20220922203457.352 ERROR    start - ../src/src/lxc/start.c:do_start:1272 - Failed to setup container "owntone"
lxc owntone 20220922203457.352 ERROR    sync - ../src/src/lxc/sync.c:sync_wait:34 - An error occurred in another process (expected sequence number 4)
lxc owntone 20220922203457.359 WARN     network - ../src/src/lxc/network.c:lxc_delete_network_priv:3631 - Failed to rename interface with index 0 from "eth0" to its initial name "veth7501d124"
lxc owntone 20220922203457.360 ERROR    lxccontainer - ../src/src/lxc/lxccontainer.c:wait_on_daemonized_start:877 - Received container state "ABORTING" instead of "RUNNING"
lxc owntone 20220922203457.360 ERROR    start - ../src/src/lxc/start.c:__lxc_start:2107 - Failed to spawn container "owntone"
lxc owntone 20220922203457.360 WARN     start - ../src/src/lxc/start.c:lxc_abort:1036 - No such process - Failed to send SIGKILL via pidfd 43 for process 782016
lxc owntone 20220922203457.460 WARN     conf - ../src/src/lxc/conf.c:lxc_map_ids:3592 - newuidmap binary is missing
lxc owntone 20220922203457.460 WARN     conf - ../src/src/lxc/conf.c:lxc_map_ids:3598 - newgidmap binary is missing
lxc 20220922203457.498 ERROR    af_unix - ../src/src/lxc/af_unix.c:lxc_abstract_unix_recv_fds_iov:218 - Connection reset by peer - Failed to receive response
lxc 20220922203457.499 ERROR    commands - ../src/src/lxc/commands.c:lxc_cmd_rsp_recv_fds:128 - Failed to receive file descriptors for command "get_state"

However, if I lxc config set owntone security.privileged true, then the container starts & runs fine.

I didn’t need to set security.privileged to true on the kernel 5.4 system… (& prior to the update of the kernel 5.15 system, i also didn’t have those containers set to use privileged mode)

One thing i noticed as different when i ran lxc info on both systems is the following kernel feature:

idmapped_mounts: “true” on the kernel 5.15 system, whereas
idmapped_mounts: “false” on the kernel 5.4 system

I noted this here because i saw another post that warns that “shiftfs shouldn’t be mounted on top of idmapped mounts …”, but (a) wasn’t sure if this is what is happening and (b) i can’t figure out how to set idmapped_mounts to false assuming that is possible and would help…

appreciate any help - i’d like to get the containers running without having to rely on privileged mode

2

i believe the issue resulted from enabling shiftfs - which isn’t needed any longer on ubuntu kernels > 5.12 if using a supported file system - on a kernel 5.15 system which has idmapped_mounts enabled.

I unset the shiftfs key in the snap, reloaded the lxd snap, unset security.privileged in the containers and then removed and readded (without the key shift=true) the folder to be shared. Now it all works as it should, except the shared folder shows up with owner:group of nobody:nogroup. So, I added a raw.idmap to the container and the mapping works now as expected.

The folder i’m sharing originates as a cifs shared folder on TrueNAS (and the underlying file system there is zfs), which is mounted as a shared folder on my lxd host. In turn, i then shared it with the container… I’m presuming the underlying zfs file system (currently not supported by idmapped_mounts) is why i needed to add the raw.idmaps and that idmapped_mounts didn’t automatically just work.

Is there a way to use shiftfs still on kernels that support idmapped_mounts and not require a privileged container?

3

Yes I’ve been experiencing the same thing as you and came to the same conclusions, see:

It seems like a recent kernel regression.

You are correct that idmapped mounts replaces shiftfs functionality, but currently doesn’t work on ZFS or TMPFS filesystems.

4

We’ve opened a bug to report the kernel regression:

5

appreciate the followup, thanks

6

Is this issue still at large? The bug report says the fix is committed to Jammy, Lunar, and Kinetic. Is that within Kernel 5.15? Is is an issue with any newer kernels as well?

I am hoping we no longer have to run containers in privileged mode if the lxd disk devices are from a zfs filesystem.

7

@amikhalitsyn did this get fixed in the ubuntu kernels yet? Thanks

8

AFAIK, the status should be “Fix Released”. So fixes are already in tree but we need to wait for the release.

1 Like
9

Thanks for the update everyone.

For anyone else following this: I just checked the bug tracker and it is released in kernel 5.15.0-69.76 which is finally available to update to.

I haven’t updated and tested yet, but this being solved means lxc containers with zfs disk devices should be able to be run without the privileged=true flag in the config again.