Strange error message while starting privileged container

Dear community.

After starting a privileged container I see the following error message in the container log.

lxc priv-test 20210809152444.594 ERROR conf - conf.c:turn_into_dependent_mounts:3724 - No such file or directory - Failed to recursively turn old root mount tree into dependent mount. Continuing...

The container seems to work without any issue.

The host is an Ubtuntu 18.04.5 LTS
5.4.0-80-generic #90~18.04.1-Ubuntu SMP Tue Jul 13 19:40:02 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

The file system is ZFS.

 lxc version
Client version: 4.16
Server version: 4.16

With an unprivileged container I see no error message, but some warnings.
That container also seems to work without any issue.

Log:

lxc no-priv-test 20210809154250.621 WARN     conf - conf.c:lxc_map_ids:3389 - newuidmap binary is missing
lxc no-priv-test 20210809154250.621 WARN     conf - conf.c:lxc_map_ids:3395 - newgidmap binary is missing
lxc no-priv-test 20210809154250.622 WARN     conf - conf.c:lxc_map_ids:3389 - newuidmap binary is missing
lxc no-priv-test 20210809154250.622 WARN     conf - conf.c:lxc_map_ids:3395 - newgidmap binary is missing
lxc no-priv-test 20210809154250.622 WARN     cgfsng - cgroups/cgfsng.c:fchowmodat:1293 - No such file or directory - Failed to fchownat(43, memory.oom.group, 1000000000, 0, AT_EMPTY_PATH | AT_SYMLINK_NOFOLLOW )

privileged container config:

lxc config show priv-test
architecture: x86_64
config:
  image.architecture: amd64
  image.description: ubuntu 18.04 LTS amd64 (release) (20210604)
  image.label: release
  image.os: ubuntu
  image.release: bionic
  image.serial: "20210604"
  image.type: squashfs
  image.version: "18.04"
  raw.lxc: lxc.init.cmd = /sbin/init systemd.unified_cgroup_hierarchy=0
  security.nesting: "true"
  security.privileged: "true"
  volatile.base_image: 682b2f9adae4a2bfefa9962cc6b2c4146fd9817ea40e5e373725baacf94fed66
  volatile.eth0.host_name: veth4e2e3c93
  volatile.eth0.hwaddr: 00:16:3e:20:93:92
  volatile.idmap.base: "0"
  volatile.idmap.current: '[]'
  volatile.idmap.next: '[]'
  volatile.last_state.idmap: '[]'
  volatile.last_state.power: RUNNING
  volatile.uuid: 85204179-182f-488c-806a-f23649d7bbdf
devices: {}
ephemeral: false
profiles:
- default
stateful: false
description: ""

cat /var/snap/lxd/common/lxd/logs/priv-test/lxc.conf

lxc.log.file = /var/snap/lxd/common/lxd/logs/priv-test/lxc.log
lxc.log.level = warn
lxc.console.buffer.size = auto
lxc.console.size = auto
lxc.console.logfile = /var/snap/lxd/common/lxd/logs/priv-test/console.log
lxc.cap.drop = sys_time sys_module sys_rawio
lxc.mount.auto = proc:mixed sys:mixed cgroup:mixed
lxc.autodev = 1
lxc.pty.max = 1024
lxc.mount.entry = mqueue dev/mqueue mqueue rw,relatime,create=dir,optional 0 0
lxc.mount.entry = /dev/fuse dev/fuse none bind,create=file,optional 0 0
lxc.mount.entry = /dev/net/tun dev/net/tun none bind,create=file,optional 0 0
lxc.mount.entry = /proc/sys/fs/binfmt_misc proc/sys/fs/binfmt_misc none rbind,create=dir,optional 0 0
lxc.mount.entry = /sys/firmware/efi/efivars sys/firmware/efi/efivars none rbind,create=dir,optional 0 0
lxc.mount.entry = /sys/fs/fuse/connections sys/fs/fuse/connections none rbind,create=dir,optional 0 0
lxc.mount.entry = /sys/fs/pstore sys/fs/pstore none rbind,create=dir,optional 0 0
lxc.mount.entry = /sys/kernel/config sys/kernel/config none rbind,create=dir,optional 0 0
lxc.mount.entry = /sys/kernel/debug sys/kernel/debug none rbind,create=dir,optional 0 0
lxc.mount.entry = /sys/kernel/security sys/kernel/security none rbind,create=dir,optional 0 0
lxc.mount.entry = /sys/kernel/tracing sys/kernel/tracing none rbind,create=dir,optional 0 0
lxc.include = /snap/lxd/current/lxc/config//common.conf.d/
lxc.cgroup.devices.deny = a
lxc.cgroup.devices.allow = b *:* m
lxc.cgroup.devices.allow = c *:* m
lxc.cgroup.devices.allow = c 136:* rwm
lxc.cgroup.devices.allow = c 1:3 rwm
lxc.cgroup.devices.allow = c 1:5 rwm
lxc.cgroup.devices.allow = c 1:7 rwm
lxc.cgroup.devices.allow = c 1:8 rwm
lxc.cgroup.devices.allow = c 1:9 rwm
lxc.cgroup.devices.allow = c 5:0 rwm
lxc.cgroup.devices.allow = c 5:1 rwm
lxc.cgroup.devices.allow = c 5:2 rwm
lxc.cgroup.devices.allow = c 10:229 rwm
lxc.cgroup.devices.allow = c 10:200 rwm
lxc.mount.entry = proc dev/.lxc/proc proc create=dir,optional 0 0
lxc.mount.entry = sys dev/.lxc/sys sysfs create=dir,optional 0 0
lxc.arch = linux64
lxc.hook.version = 1
lxc.hook.pre-start = /proc/16070/exe callhook /var/snap/lxd/common/lxd "default" "priv-test" start
lxc.hook.stop = /snap/lxd/current/bin/lxd callhook /var/snap/lxd/common/lxd "default" "priv-test" stopns
lxc.hook.post-stop = /snap/lxd/current/bin/lxd callhook /var/snap/lxd/common/lxd "default" "priv-test" stop
lxc.tty.max = 0
lxc.uts.name = priv-test
lxc.mount.entry = /var/snap/lxd/common/lxd/devlxd dev/lxd none bind,create=dir 0 0
lxc.apparmor.profile = lxd-priv-test_</var/snap/lxd/common/lxd>//&:lxd-priv-test_<var-snap-lxd-common-lxd>:
lxc.seccomp.profile = /var/snap/lxd/common/lxd/security/seccomp/priv-test
lxc.mount.auto = shmounts:/var/snap/lxd/common/lxd/shmounts/priv-test:/dev/.lxd-mounts
lxc.init.cmd = /sbin/init systemd.unified_cgroup_hierarchy=0
lxc.net.0.type = phys
lxc.net.0.name = eth0
lxc.net.0.flags = up
lxc.net.0.link = veth746e2e70
lxc.rootfs.path = dir:/var/snap/lxd/common/lxd/containers/priv-test/rootfs

unprivileged container config

lxc config show no-priv-test
architecture: x86_64
config:
  image.architecture: amd64
  image.description: ubuntu 18.04 LTS amd64 (release) (20210604)
  image.label: release
  image.os: ubuntu
  image.release: bionic
  image.serial: "20210604"
  image.type: squashfs
  image.version: "18.04"
  volatile.base_image: 682b2f9adae4a2bfefa9962cc6b2c4146fd9817ea40e5e373725baacf94fed66
  volatile.eth0.host_name: veth19693825
  volatile.eth0.hwaddr: 00:16:3e:d4:d4:dc
  volatile.idmap.base: "0"
  volatile.idmap.current: '[{"Isuid":true,"Isgid":false,"Hostid":1000000,"Nsid":0,"Maprange":1000000000},{"Isuid":false,"Isgid":true,"Hostid":1000000,"Nsid":0,"Maprange":1000000000}]'
  volatile.idmap.next: '[{"Isuid":true,"Isgid":false,"Hostid":1000000,"Nsid":0,"Maprange":1000000000},{"Isuid":false,"Isgid":true,"Hostid":1000000,"Nsid":0,"Maprange":1000000000}]'
  volatile.last_state.idmap: '[{"Isuid":true,"Isgid":false,"Hostid":1000000,"Nsid":0,"Maprange":1000000000},{"Isuid":false,"Isgid":true,"Hostid":1000000,"Nsid":0,"Maprange":1000000000}]'
  volatile.last_state.power: RUNNING
  volatile.uuid: 58827c93-e112-4d01-ad85-fe880b139bd9
devices: {}
ephemeral: false
profiles:
- default
stateful: false
description: ""

 cat /var/snap/lxd/common/lxd/logs/no-priv-test/lxc.conf
lxc.log.file = /var/snap/lxd/common/lxd/logs/no-priv-test/lxc.log
lxc.log.level = warn
lxc.console.buffer.size = auto
lxc.console.size = auto
lxc.console.logfile = /var/snap/lxd/common/lxd/logs/no-priv-test/console.log
lxc.mount.auto = proc:rw sys:rw cgroup:mixed
lxc.autodev = 1
lxc.pty.max = 1024
lxc.mount.entry = /dev/fuse dev/fuse none bind,create=file,optional 0 0
lxc.mount.entry = /dev/net/tun dev/net/tun none bind,create=file,optional 0 0
lxc.mount.entry = /proc/sys/fs/binfmt_misc proc/sys/fs/binfmt_misc none rbind,create=dir,optional 0 0
lxc.mount.entry = /sys/firmware/efi/efivars sys/firmware/efi/efivars none rbind,create=dir,optional 0 0
lxc.mount.entry = /sys/fs/fuse/connections sys/fs/fuse/connections none rbind,create=dir,optional 0 0
lxc.mount.entry = /sys/fs/pstore sys/fs/pstore none rbind,create=dir,optional 0 0
lxc.mount.entry = /sys/kernel/config sys/kernel/config none rbind,create=dir,optional 0 0
lxc.mount.entry = /sys/kernel/debug sys/kernel/debug none rbind,create=dir,optional 0 0
lxc.mount.entry = /sys/kernel/security sys/kernel/security none rbind,create=dir,optional 0 0
lxc.mount.entry = /sys/kernel/tracing sys/kernel/tracing none rbind,create=dir,optional 0 0
lxc.mount.entry = /dev/mqueue dev/mqueue none rbind,create=dir,optional 0 0
lxc.include = /snap/lxd/current/lxc/config//common.conf.d/
lxc.arch = linux64
lxc.hook.version = 1
lxc.hook.pre-start = /proc/16070/exe callhook /var/snap/lxd/common/lxd "default" "no-priv-test" start
lxc.hook.stop = /snap/lxd/current/bin/lxd callhook /var/snap/lxd/common/lxd "default" "no-priv-test" stopns
lxc.hook.post-stop = /snap/lxd/current/bin/lxd callhook /var/snap/lxd/common/lxd "default" "no-priv-test" stop
lxc.tty.max = 0
lxc.uts.name = no-priv-test
lxc.mount.entry = /var/snap/lxd/common/lxd/devlxd dev/lxd none bind,create=dir 0 0
lxc.apparmor.profile = lxd-no-priv-test_</var/snap/lxd/common/lxd>//&:lxd-no-priv-test_<var-snap-lxd-common-lxd>:
lxc.seccomp.profile = /var/snap/lxd/common/lxd/security/seccomp/no-priv-test
lxc.idmap = u 0 1000000 1000000000
lxc.idmap = g 0 1000000 1000000000
lxc.mount.auto = shmounts:/var/snap/lxd/common/lxd/shmounts/no-priv-test:/dev/.lxd-mounts
lxc.net.0.type = phys
lxc.net.0.name = eth0
lxc.net.0.flags = up
lxc.net.0.link = veth134368ae
lxc.rootfs.path = dir:/var/snap/lxd/common/lxd/containers/no-priv-test/rootfs

@brauner

I tried a few things to get rid of the problem.

Our lxd/lxc has been installed via snap

patched the lxd/lxc to version 4.17 → no change
disabled apparmor for testing → no change

installed uidmap via apt to get rid of warnings:

" … conf.c:lxc_map_ids:3389 - newuidmap binary is missing … "
“… conf.c:lxc_map_ids:3395 - newgidmap binary is missing …”

So the binaries exist now:
/usr/bin/newuidmap
/usr/bin/newgidmap

but no change, the unprivileged container still shows the warnings in its log.

I created some soflinks:

/bin/newuidmap
/bin/newgidmap

/sbin/newuidmap
/sbin/newgidmap

but still no change.