Cannot restart container after remapping user id

1

Hello everyone,

I am trying to map a directory from the host to a container with read and write permissions and was following this example https://www.cyberciti.biz/faq/how-to-add-or-mount-directory-in-lxd-linux-container/ . However, after remapping my user id i cannot restart my container. I’m not sure what i’m doing wrong.

This is the error message:

Error: Common start logic: invalid argument - Failed to change ACLs on /var/snap/lxd/common/lxd/storage-pools/pool-name/containers/container-name/rootfs/var/log/journal

Try lxc info --show-log container-name for more info

and the log of lxc info --show-log container-name:

Log:

lxc container-name 20201116144756.467 WARN cgfsng - cgroups/cgfsng.c:mkdir_eexist_on_last:1152 - File exists - Failed to create directory “/sys/fs/cgroup/cpuset//lxc.monitor.container-name”
lxc container-name 20201116144756.468 WARN cgfsng - cgroups/cgfsng.c:mkdir_eexist_on_last:1152 - File exists - Failed to create directory "/sys/fs/cgroup/cpuset//lxc.payload.container name
lxc container-name 20201116144756.470 ERROR utils - utils.c:lxc_can_use_pidfd:1846 - Kernel does not support pidfds
lxc container-name 20201116144801.163 WARN utils - utils.c:fix_stdio_permissions:1906 - Operation not permitted - Failed to chown standard I/O file descriptor 0 to uid -1 and gid 65534
lxc container-name 20201116144801.163 WARN utils - utils.c:fix_stdio_permissions:1912 - Operation not permitted - Failed to chmod standard I/O file descriptor 0
lxc container-name 20201116144801.163 WARN utils - utils.c:fix_stdio_permissions:1906 - Operation not permitted - Failed to chown standard I/O file descriptor 1 to uid -1 and gid 65534
lxc container-name 20201116144801.163 WARN utils - utils.c:fix_stdio_permissions:1912 - Operation not permitted - Failed to chmod standard I/O file descriptor 1
lxc container-name 20201116144801.163 WARN utils - utils.c:fix_stdio_permissions:1906 - Operation not permitted - Failed to chown standard I/O file descriptor 2 to uid -1 and gid 65534
lxc container-name 20201116144801.163 WARN utils - utils.c:fix_stdio_permissions:1912 - Operation not permitted - Failed to chmod standard I/O file descriptor 2
lxc container-name 20201116144801.163 WARN attach - attach.c:attach_child_main:882 - Failed to adjust stdio permissions
lxc container-name 20201116153501.392 ERROR commands - commands.c:lxc_cmd_get_init_pidfd_callback:457 - Failed to send init pidfd

I am running lxd version 4.7 on btrfs and kernel is 4.15.0.

Thanks in advance for your help.

2

If you can, delete that path from your instance, that will fix it.

We’ve had issues with id shifting of the systemd journal in the past, those issues have been resolved but won’t do you much good with an existing container unfortunately.

So the best option is to delete the path, that should clear the issue and all you lose is your journal log entries which are mostly replicated in log files anyway if need be.

3

Thanks for your quick reply @stgraber. It looks like i do not have that path, i tried to check my container as well in /var/snap/lxd/common/lxd/storage-pools/Pool-1/containers/ and there’s nothing listed.

ll /var/snap/lxd/common/lxd/storage-pools/pool-name/

total 8
drwx–x--x 2 root root 4096 Nov 3 14:48 ./
drwx–x--x 3 root root 4096 Nov 3 14:48 …/

This is also what i have in my lxc config:

architecture: x86_64
config:
image.architecture: amd64
image.description: ubuntu 18.04 LTS amd64 (release) (20201031)
image.label: release
image.os: ubuntu
image.release: bionic
image.serial: “20201031”
image.type: squashfs
image.version: “18.04”
raw.idmap: both 1002 1002
volatile.base_image: 6063f2011ba10d2dfdb2518ae02a1ddb621334a002bdf3def111dfd54dce2abd
volatile.eth0.hwaddr: 00:16:3e:79:6b:7f
volatile.idmap.base: “0”
volatile.idmap.current: ‘[{“Isuid”:true,“Isgid”:false,“Hostid”:1000000,“Nsid”:0,“Maprange”:1000000000},{“Isuid”:false,“Isgid”:true,“Hostid”:1000000,“Nsid”:0,“Maprange”:1000000000}]’
volatile.idmap.next: ‘[{“Isuid”:true,“Isgid”:false,“Hostid”:1000000,“Nsid”:0,“Maprange”:1002},{“Isuid”:true,“Isgid”:true,“Hostid”:1002,“Nsid”:1002,“Maprange”:1},{“Isuid”:true,“Isgid”:false,“Hostid”:1001003,“Nsid”:1003,“Maprange”:999998997},{“Isuid”:false,“Isgid”:true,“Hostid”:1000000,“Nsid”:0,“Maprange”:1002},{“Isuid”:true,“Isgid”:true,“Hostid”:1002,“Nsid”:1002,“Maprange”:1},{“Isuid”:false,“Isgid”:true,“Hostid”:1001003,“Nsid”:1003,“Maprange”:999998997}]’
volatile.last_state.idmap: ‘[{“Isuid”:true,“Isgid”:false,“Hostid”:1000000,“Nsid”:0,“Maprange”:1000000000},{“Isuid”:false,“Isgid”:true,“Hostid”:1000000,“Nsid”:0,“Maprange”:1000000000}]’
volatile.last_state.power: STOPPED
devices:
eth0:
name: eth0
nictype: bridged
parent: br0
type: nic
root:
path: /
pool: pool-name
type: disk
ephemeral: false
profiles:

  • default
    stateful: false
    description: “”
4

After running lxc config unset container-name raw.idmap i can succesfully start the container. In this case, what would be the way to map a directory into my container with read/write mode since the guided i followed did not work?

5

Once you get the container back online, wipe /var/log/journal, stop the container and then try applying your config again, that should get you past the remap issue.

6

I tried that one but it looks like my container is located in:

ll /var/snap/lxd/common/lxd/containers/container-name

but the same error: Error: Common start logic: invalid argument - Failed to change ACLs on /var/snap/lxd/common/lxd/storage-pools/pool-name/containers/container-name/rootfs/var/log/journal

It’s trying to find that path but as mentioned before there’s nothing in this path:

ll /var/snap/lxd/common/lxd/storage-pools/pool-name/
total 8
drwx–x--x 2 root root 4096 Nov 3 14:48 ./
drwx–x--x 3 root root 4096 Nov 3 14:48 …/

And also if i do

ls -lh /var/snap/lxd/common/mntns/var/snap/lxd/common/lxd/storage-pools/pool-name/containers/container-name/rootfs/

total 0
drwxr-xr-x 1 root root 2.4K Nov 4 15:13 bin
drwxr-xr-x 1 root root 0 Oct 31 12:11 boot
drwxr-xr-x 1 root root 114 Oct 31 12:02 dev
drwxr-xr-x 1 root root 3.2K Nov 16 10:32 etc
drwxr-xr-x 1 root root 34 Nov 16 10:31 home
drwxr-xr-x 1 root root 438 Oct 31 12:00 lib
drwxr-xr-x 1 root root 40 Nov 4 15:13 lib64
drwxr-xr-x 1 root root 0 Oct 31 11:53 media
drwxr-xr-x 1 root root 0 Oct 31 11:53 mnt
drwxr-xr-x 1 root root 0 Oct 31 11:53 opt
drwxr-xr-x 1 root root 0 Apr 24 2018 proc
drwx------ 1 root root 122 Nov 13 14:14 root
drwxr-xr-x 1 root root 16 Oct 31 12:11 run
drwxr-xr-x 1 root root 3.7K Nov 4 15:13 sbin
drwxr-xr-x 1 root root 12 Nov 3 15:19 snap
drwxr-xr-x 1 root root 0 Oct 31 11:53 srv
drwxr-xr-x 1 root root 0 Apr 24 2018 sys
drwxrwxrwt 1 root root 94 Nov 16 15:04 tmp
drwxr-xr-x 1 root root 70 Oct 31 11:53 usr
drwxr-xr-x 1 root root 108 Oct 31 12:03 var

I can provide more info if needed.

Thanks again!

7

You said you had the container running earlier, so just get it running again, lxc exec NAME bash and then run rm -rf /var/log/journal from within the container.

8

Yes, i did get the container running again but when i run rm -rf /var/log/journal within the container:

rm: cannot remove ‘/var/log/journal/e546d47f28894237ada3e9d623ed286f’: Permission denied

and when i do:

ll /var/log/ half of my files are own by nobody:nogroup and all my files in my /root/ directory are also own by nobody:nogroup.

I also tried this solution as well but my files are still owned by nobody:nogroup.

9

Ah, okay, so you’re dealing with a partly shifted container…

While the container is running, you should be able to wipe the directory through /var/snap/lxd/common/mntns/var/snap/lxd/common/lxd/storage-pools/POOL-NAME/containers/NAME/rootfs/var/log/journal. Then go back to changing config and restarting the container

10

Thanks, i really appreciate your help @stgraber. I had to start from the scratch following your documentation here. https://ubuntu.com/blog/custom-user-mappings-in-lxd-containers and this helped me to do it in the right way. I had to set a custom map and that did the trick.

Thanks again!

1 Like