User "nobody" and group "nogroup" after the migration of the containers

Hello!

I need to migrate several containers from host-a to host-b.

  • Host-a : LXC 2.0.7 (Debian 9)
  • Host-b : LXC 3.0.3 (Ubuntu 18.04.3 LTS)

From host-a, I rsynced the containers to a temporary directory on host-b :

rsync --archive --delete-after /home/containers/ host-b:/home/host-a-containers

From host-b, I created fresh containers, and replaced their rootfs content with the rootfs from the temporary directory.

lxc launch images:debian/jessie container1
lxc stop container1
rm -rf /home/containers/container1/rootfs
mv /home/host-a-containers/container1/rootfs /home/containers/container1/
lxc start container1

The containers are working, but when I get a shell inside I have an ownership problem :

root@host-b:~# lxc exec container1 -- /bin/bash
bash: /root/.bashrc: Permission denied
root@container1:~# ls
ls: cannot open directory .: Permission denied
root@container1:~# cd /
root@container1:/# ls -l
total 64
drwxr-xr-x   2 nobody nogroup 4096 Sep 16  2018 bin
drwxr-xr-x   2 nobody nogroup 4096 Mar 14  2016 boot
drwxr-xr-x   8 root   root     440 Dec  5 15:56 dev
drwxr-xr-x  55 nobody nogroup 4096 Jan 13  2019 etc
drwxr-xr-x   3 nobody nogroup 4096 Jun 11  2017 home
drwxr-xr-x  10 nobody nogroup 4096 May 28  2016 lib
drwxr-xr-x   2 nobody nogroup 4096 Sep 16  2018 lib64
drwxr-xr-x   2 nobody nogroup 4096 May 28  2016 media
drwxr-xr-x   2 nobody nogroup 4096 May 28  2016 mnt
drwxr-xr-x   2 nobody nogroup 4096 May 28  2016 opt
dr-xr-xr-x 199 nobody nogroup    0 Dec  5 15:56 proc
drwx------   8 nobody nogroup 4096 Dec  5 14:24 root
drwxr-xr-x   4 root   root      80 Dec  5 15:56 run
drwxr-xr-x   2 nobody nogroup 4096 Sep 16  2018 sbin
drwxr-xr-x   2 nobody nogroup 4096 May 28  2016 selinux
drwxr-xr-x   2 nobody nogroup 4096 May 28  2016 srv
dr-xr-xr-x  13 nobody nogroup    0 Dec  5 15:44 sys
drwxrwxrwt   7 nobody nogroup 4096 Dec  5 14:22 tmp
drwxr-xr-x  10 nobody nogroup 4096 May 28  2016 usr
drwxr-xr-x  12 nobody nogroup 4096 May 28  2016 var

It seems all UID and GID in the containers are offset by 65 534:

root@container1:/# ls -n
total 64
drwxr-xr-x   2 65534 65534 4096 Sep 16  2018 bin
drwxr-xr-x   2 65534 65534 4096 Mar 14  2016 boot
drwxr-xr-x   9     0     0  500 Dec  5 15:57 dev
drwxr-xr-x  55 65534 65534 4096 Jan 13  2019 etc
drwxr-xr-x   3 65534 65534 4096 Jun 11  2017 home
drwxr-xr-x  10 65534 65534 4096 May 28  2016 lib
drwxr-xr-x   2 65534 65534 4096 Sep 16  2018 lib64
drwxr-xr-x   2 65534 65534 4096 May 28  2016 media
drwxr-xr-x   2 65534 65534 4096 May 28  2016 mnt
drwxr-xr-x   2 65534 65534 4096 May 28  2016 opt
dr-xr-xr-x 207 65534 65534    0 Dec  5 15:56 proc
drwx------   8 65534 65534 4096 Dec  5 14:24 root
drwxr-xr-x  10     0     0  280 Dec  5 15:57 run
drwxr-xr-x   2 65534 65534 4096 Sep 16  2018 sbin
drwxr-xr-x   2 65534 65534 4096 May 28  2016 selinux
drwxr-xr-x   2 65534 65534 4096 May 28  2016 srv
dr-xr-xr-x  13 65534 65534    0 Dec  5 15:44 sys
drwxrwxrwt   7 65534 65534 4096 Dec  5 15:57 tmp
drwxr-xr-x  10 65534 65534 4096 May 28  2016 usr
drwxr-xr-x  12 65534 65534 4096 May 28  2016 var

I can run the following command from the host to fix some directories, but I can’t use it for all of them :

chown -R 100000:100000 /home/containers/container1/rootfs/root

What have I done wrong?
Is there a better way to proceed without updating LXC on host-a?

Thank you :slight_smile:

Blaise

This could be because you dont have the same subid and subgid settings on the new host.

1 Like

I think you are right, thank you for your help :blush:

On host-a:

root@host-a:~# cat /etc/subuid
systemd-timesync:100000:65536
systemd-network:165536:65536
systemd-resolve:231072:65536
systemd-bus-proxy:296608:65536
sshd:362144:65536
bind:493216:65536
ntp:427680:65536
root@host-a:~# cat /etc/subgid
systemd-timesync:100000:65536
systemd-network:165536:65536
systemd-resolve:231072:65536
systemd-bus-proxy:296608:65536
sshd:362144:65536
bind:493216:65536
ntp:427680:65536

On host-b:

root@host-b:~# cat /etc/subuid
lxd:100000:65536
root:100000:65536
root@host-b:~# cat /etc/subgid
lxd:100000:65536
root:100000:65536

What can I do to solve my nobody and nogroup problem, if I want to keep my containers unprivileged on host-b and if I don’t want to risk breaking anything on host-a which is in production?

I just find a usefull option for the chown command : --from=CURRENT_OWNER:CURRENT_GROUP

This option allow to change the owner and/or group of each file only if its current owner and/or group match those specified.

So I can get the UID/GID from the container’s /etc/shadow file, add 100 000, and change the ownership of all files from the host.

root@host-b:~# awk -F: '{printf "chown -R --from=%s:%s %s:%s /home/containers/container1/rootfs\n",$3,$4,($3+100000),($4+100000)}' /home/containers/container1/rootfs/etc/passwd
chown -R --from=0:0 100000:100000 /home/containers/container1/rootfs
chown -R --from=1:1 100001:100001 /home/containers/container1/rootfs
chown -R --from=2:2 100002:100002 /home/containers/container1/rootfs
chown -R --from=3:3 100003:100003 /home/containers/container1/rootfs
chown -R --from=4:65534 100004:165534 /home/containers/container1/rootfs
chown -R --from=5:60 100005:100060 /home/containers/container1/rootfs
chown -R --from=6:12 100006:100012 /home/containers/container1/rootfs
chown -R --from=7:7 100007:100007 /home/containers/container1/rootfs
chown -R --from=8:8 100008:100008 /home/containers/container1/rootfs
chown -R --from=9:9 100009:100009 /home/containers/container1/rootfs
chown -R --from=10:10 100010:100010 /home/containers/container1/rootfs
chown -R --from=13:13 100013:100013 /home/containers/container1/rootfs
chown -R --from=33:33 100033:100033 /home/containers/container1/rootfs
chown -R --from=34:34 100034:100034 /home/containers/container1/rootfs
chown -R --from=38:38 100038:100038 /home/containers/container1/rootfs
chown -R --from=39:39 100039:100039 /home/containers/container1/rootfs
chown -R --from=41:41 100041:100041 /home/containers/container1/rootfs
chown -R --from=65534:65534 165534:165534 /home/containers/container1/rootfs
chown -R --from=100:103 100100:100103 /home/containers/container1/rootfs
chown -R --from=101:104 100101:100104 /home/containers/container1/rootfs
chown -R --from=102:105 100102:100105 /home/containers/container1/rootfs
chown -R --from=103:106 100103:100106 /home/containers/container1/rootfs
chown -R --from=104:65534 100104:165534 /home/containers/container1/rootfs

It seems to work, but… It’s not very elegant…

EDIT : it doesn’t work as expected, see below

If someone can explain to me what is the right way to solve this subuid/gid, it would be very appreciated :blush:

Basically you should just add the lines from the original host to the new one:

However I notice that some of your existing maps on the new host overlap with the ID ranges from the old host, albeit with different users, so it may be fine, but I’m not sure.

@stgraber can you have multiple users with overlapping ID ranges?

Although looking more closely, you don’t appear to have an lxc subuid/subgid map on your old host, was this a privileged container?

If so, then using the new uid map on host b and doing the chown trick you found should work.

Yes, the old host (host-a) has no lxd/root entry in subuid/subgid.

Some of the containers were created in 2013, then updated, and migrated several times since. I don’t even know if it was possible to create unprivileged containers at the time.

If someone else try to migrate containers with rsync, you MUST use the --numeric-ids option, because otherwise, some IDs change during the transfer!

rsync --numeric-ids --archive --delete-after /home/containers/ host-b:/home/host-a-containers

Also, my previous trick to convert the subuid and subgid is wrong, because some groups have no users. So it is necessary to get and convert the list of users and groups separately:

For users:
awk -F: '{printf "chown -R --from=%s %s /home/containers/container1/rootfs &&\n",$3,($3+100000)}' /home/containers/container1/rootfs/etc/passwd

For groups:
awk -F: '{printf "chown -R --from=:%s :%s /home/containers/container1/rootfs &&\n",$3,($3+100000)}' /home/containers/container1/rootfs/etc/group