Starting any container results in newuidmap errors: "Target process [pid] is owned by a different user: uid:0[...]"

  • Distribution: Gentoo ~amd64
  • lxc-start --version: 4.0.12
  • lxc-checkconfig: dpaste: ALQHG3CGS
  • uname -a: Linux desktop 5.16.9-gentoo #1 SMP PREEMPT Tue Feb 15 10:18:31 NZDT 2022 x86_64 AMD Ryzen 7 3700X 8-Core Processor AuthenticAMD GNU/Linux
  • cat /proc/self/cgroup: 0::/user.slice/user-1000.slice/user@1000.service/app.slice/app-org.kde.konsole-784e1481bbd94ba4ae220ec59df35793.scope
  • cat /proc/1/mounts: https://dpaste.com/CQNJXD36S

Issue description

Starting a container results in the following error related to new{uid,gid}map:

lxc gentoo-base 20220216082112.267 ERROR    conf - conf.c:lxc_map_ids:3668 - newuidmap failed to write mapping "newuidmap: Target process 188239 is owned by a different user: uid:0 pw_uid:0 st_uid:0, gid:402 pw_gid:0 st_gid:402": newuidmap 188239 0 1000000 1000000000
lxc gentoo-base 20220216082112.267 ERROR    start - start.c:lxc_spawn:1791 - Failed to set up id mapping.
lxc gentoo-base 20220216082112.267 ERROR    lxccontainer - lxccontainer.c:wait_on_daemonized_start:878 - Received container state "ABORTING" instead of "RUNNING"
lxc gentoo-base 20220216082112.267 ERROR    start - start.c:__lxc_start:2074 - Failed to spawn container "gentoo-base"
lxc gentoo-base 20220216082112.267 WARN     start - start.c:lxc_abort:1040 - No such process - Failed to send SIGKILL via pidfd 17 for process 188239

LXD is run as a systemd service as user root and group lxd. My user is in the lxd group.
This is strange as according from what I’ve read online the /etc/sub{u,g}id files are correctly set. Output of them both:

telans@desktop ~ $ cat /etc/sub{u,g}id
root:1000000:1000000000
lxd:1000000:1000000000
telans:1001000000:1000000
root:1000000:1000000000
lxd:1000000:1000000000
telans:1001000000:1000000

The permissions of the newmap binaries do not look wrong:

-rws--x--x 1 root root 51K Feb 16 16:05 /usr/bin/newgidmap
-rws--x--x 1 root root 50K Feb 16 16:05 /usr/bin/newuidmap

When moving the binaries out of the way (.bak etc) or by removing the executable flag the containers start up warning about missing the binaries, however, the container uid’s are set correctly in the range specified in /etc/subuid.

I cannot figure out why newuidmap appears to be failing. Any ideas?

Steps to reproduce

Unknown.

Information to attach

  • [x] container log: Not running as unprivileged user, lxc info --show-log gentoo-base:
Name: gentoo-base
Status: STOPPED
Type: container
Architecture: x86_64
Created: 2022/02/16 21:00 NZDT
Last Used: 2022/02/16 21:21 NZDT

Log:

lxc gentoo-base 20220216082112.248 WARN     cgfsng - cgroups/cgfsng.c:__cgroup_tree_create:735 - File exists - Creating the final cgroup 10(lxc.payload.gentoo-base) failed
lxc gentoo-base 20220216082112.249 WARN     cgfsng - cgroups/cgfsng.c:cgroup_tree_create:795 - File exists - Failed to create payload cgroup 10(lxc.payload.gentoo-base)
lxc gentoo-base 20220216082112.249 WARN     cgfsng - cgroups/cgfsng.c:__cgroup_tree_create:735 - File exists - Creating the final cgroup 10(lxc.payload.gentoo-base-1) failed
lxc gentoo-base 20220216082112.249 WARN     cgfsng - cgroups/cgfsng.c:cgroup_tree_create:795 - File exists - Failed to create payload cgroup 10(lxc.payload.gentoo-base-1)
lxc gentoo-base 20220216082112.249 WARN     cgfsng - cgroups/cgfsng.c:__cgroup_tree_create:735 - File exists - Creating the final cgroup 10(lxc.payload.gentoo-base-2) failed
lxc gentoo-base 20220216082112.249 WARN     cgfsng - cgroups/cgfsng.c:cgroup_tree_create:795 - File exists - Failed to create payload cgroup 10(lxc.payload.gentoo-base-2)
lxc gentoo-base 20220216082112.249 WARN     cgfsng - cgroups/cgfsng.c:__cgroup_tree_create:735 - File exists - Creating the final cgroup 10(lxc.payload.gentoo-base-3) failed
lxc gentoo-base 20220216082112.249 WARN     cgfsng - cgroups/cgfsng.c:cgroup_tree_create:795 - File exists - Failed to create payload cgroup 10(lxc.payload.gentoo-base-3)
lxc gentoo-base 20220216082112.249 WARN     cgfsng - cgroups/cgfsng.c:__cgroup_tree_create:735 - File exists - Creating the final cgroup 10(lxc.payload.gentoo-base-4) failed
lxc gentoo-base 20220216082112.249 WARN     cgfsng - cgroups/cgfsng.c:cgroup_tree_create:795 - File exists - Failed to create payload cgroup 10(lxc.payload.gentoo-base-4)
lxc gentoo-base 20220216082112.249 WARN     cgfsng - cgroups/cgfsng.c:__cgroup_tree_create:735 - File exists - Creating the final cgroup 10(lxc.payload.gentoo-base-5) failed
lxc gentoo-base 20220216082112.249 WARN     cgfsng - cgroups/cgfsng.c:cgroup_tree_create:795 - File exists - Failed to create payload cgroup 10(lxc.payload.gentoo-base-5)
lxc gentoo-base 20220216082112.249 WARN     cgfsng - cgroups/cgfsng.c:__cgroup_tree_create:735 - File exists - Creating the final cgroup 10(lxc.payload.gentoo-base-6) failed
lxc gentoo-base 20220216082112.249 WARN     cgfsng - cgroups/cgfsng.c:cgroup_tree_create:795 - File exists - Failed to create payload cgroup 10(lxc.payload.gentoo-base-6)
lxc gentoo-base 20220216082112.249 WARN     cgfsng - cgroups/cgfsng.c:__cgroup_tree_create:735 - File exists - Creating the final cgroup 10(lxc.payload.gentoo-base-7) failed
lxc gentoo-base 20220216082112.249 WARN     cgfsng - cgroups/cgfsng.c:cgroup_tree_create:795 - File exists - Failed to create payload cgroup 10(lxc.payload.gentoo-base-7)
lxc gentoo-base 20220216082112.267 ERROR    conf - conf.c:lxc_map_ids:3668 - newuidmap failed to write mapping "newuidmap: Target process 188239 is owned by a different user: uid:0 pw_uid:0 st_uid:0, gid:402 pw_gid:0 st_gid:402": newuidmap 188239 0 1000000 1000000000
lxc gentoo-base 20220216082112.267 ERROR    start - start.c:lxc_spawn:1791 - Failed to set up id mapping.
lxc gentoo-base 20220216082112.267 ERROR    lxccontainer - lxccontainer.c:wait_on_daemonized_start:878 - Received container state "ABORTING" instead of "RUNNING"
lxc gentoo-base 20220216082112.267 ERROR    start - start.c:__lxc_start:2074 - Failed to spawn container "gentoo-base"
lxc gentoo-base 20220216082112.267 WARN     start - start.c:lxc_abort:1040 - No such process - Failed to send SIGKILL via pidfd 17 for process 188239
lxc gentoo-base 20220216082117.273 WARN     cgfsng - cgroups/cgfsng.c:cgroup_tree_remove:483 - Permission denied - Failed to destroy 10(lxc.payload.gentoo-base-8)
lxc 20220216082117.286 ERROR    af_unix - af_unix.c:lxc_abstract_unix_recv_fds_iov:218 - Connection reset by peer - Failed to receive response
lxc 20220216082117.287 ERROR    commands - commands.c:lxc_cmd_rsp_recv_fds:127 - Failed to receive file descriptors for command "get_state"
  • [x] the containers configuration file:
architecture: x86_64
config:
  image.architecture: amd64
  image.description: Gentoo current amd64 (20220215_16:10)
  image.os: Gentoo
  image.release: current
  image.requirements.secureboot: "false"
  image.serial: "20220215_16:10"
  image.type: squashfs
  image.variant: openrc
  volatile.base_image: 62d1a3734522aa940e8f9f648cd34b8408660aab0000dca25ddfda62ceb0cade
  volatile.idmap.base: "0"
  volatile.idmap.current: '[{"Isuid":true,"Isgid":false,"Hostid":1000000,"Nsid":0,"Maprange":1000000000},{"Isuid":false,"Isgid":true,"Hostid":1000000,"Nsid":0,"Maprange":1000000000}]'
  volatile.idmap.next: '[{"Isuid":true,"Isgid":false,"Hostid":1000000,"Nsid":0,"Maprange":1000000000},{"Isuid":false,"Isgid":true,"Hostid":1000000,"Nsid":0,"Maprange":1000000000}]'
  volatile.last_state.idmap: '[]'
  volatile.last_state.power: STOPPED
  volatile.uuid: 04d0ad86-6c12-4389-8172-90267250d4ab
devices: {}
ephemeral: false
profiles:
- default
stateful: false
description: ""

newuidmap doesn’t seem to play nice when the lxd systemd service is started as user root and group lxd.

The service file had:

User=root
Group=lxd

Removing the Group line solves the issue.