Unprivileged lxc mount zfs dataset

gusto · July 8, 2021, 3:56pm

I use proxmox 6.4-9. I installed unprivileged LXC where I run nextcloud. Everything seems to be working great.
lxc. config

arch: amd64
cores: 2
hostname: web-debian-test
memory: 2048
net0: name=eth0,bridge=vmbr1,gw=192.168.1.1,hwaddr=26:40:D5:E4:61:86,ip=192.168.1.108/24,type=veth
ostype: debian
rootfs: local-zfs:subvol-108-disk-0,size=16G
swap: 2048
unprivileged: 1

On the SSD, I created a zpool called a datapool. In zpool, I created a 200 GB nextcloud dataset

# zfs list
NAME                           USED  AVAIL     REFER  MOUNTPOINT
datapool                      3.14M   899G       96K  /datapool
datapool/nextcloud             104K   200G      104K  /datapool/nextcloud
rpool                         35.3G   193G       96K  /rpool
rpool/ROOT                    24.7G   193G       96K  /rpool/ROOT
...

Before mount, I check the owners and groups inside the LXC

$ ls -la /var/www
total 26
drwxr-xr-x  4 root     root      4 Jun  7 15:51 .
drwxr-xr-x 12 root     root     14 Jun  7 14:26 ..
drwxr-xr-x  4 root     root      5 Jun  7 15:23 html
drwxrwx---  4 www-data www-data  8 Jun  7 15:55 nextcloud-data

$ ls -la /var/www/html
total 35
drwxr-xr-x  4 root     root         5 Jun  7 15:23 .
drwxr-xr-x  4 root     root         4 Jun  7 15:51 ..
-rw-r--r--  1 root     root     10701 Jun  7 14:26 index.html
drwxr-xr-x 13 www-data www-data    31 Jun  7 15:02 phpmyadmin
drwxr-xr-x 14 www-data www-data    28 Jul  3 01:17 www.nextcloud.ddns.info

sudo ls -la /var/www/nextcloud-data
total 28
drwxrwx---  4 www-data www-data   8 Jun  7 15:55 .
drwxr-xr-x  4 root     root       4 Jun  7 15:51 ..
-rw-r--r--  1 www-data www-data 542 Jun  7 15:54 .htaccess
-rw-r--r--  1 www-data www-data   0 Jun  7 15:54 .ocdata
drwxr-xr-x 10 www-data www-data  10 Jul  2 15:21 appdata_oc65vcb3bp4d
-rw-r--r--  1 www-data www-data   0 Jun  7 15:54 index.html
drwxr-xr-x  4 www-data www-data   4 Jun  7 19:27 nextcloud
-rw-r-----  1 www-data www-data 391 Jun  7 17:30 nextcloud.log

On host proxmox

ls -la /rpool/data/subvol-108-disk-1/var/www/html
total 35
drwxr-xr-x  4 100000 100000     5 Jun  7 15:23 .
drwxr-xr-x  4 100000 100000     4 Jun  7 15:51 ..
-rw-r--r--  1 100000 100000 10701 Jun  7 14:26 index.html
drwxr-xr-x 13 100033 100033    31 Jun  7 15:02 phpmyadmin
drwxr-xr-x 14 100033 100033    28 Jul  3 01:17 www.nextcloud.ddns.info

The contents of the directory intended for the data

ls -la /rpool/data/subvol-108-disk-1/var/www/nextcloud-data
total 28
drwxrwx---  4 100033 100033   8 Jun  7 15:55 .
drwxr-xr-x  4 100000 100000   4 Jun  7 15:51 ..
drwxr-xr-x 10 100033 100033  10 Jul  2 15:21 appdata_oc65vcb3bp4d
-rw-r--r--  1 100033 100033 542 Jun  7 15:54 .htaccess
-rw-r--r--  1 100033 100033   0 Jun  7 15:54 index.html
drwxr-xr-x  4 100033 100033   4 Jun  7 19:27 nextcloud
-rw-r-----  1 100033 100033 391 Jun  7 17:30 nextcloud.log
-rw-r--r--  1 100033 100033   0 Jun  7 15:54 .ocdata

Now i will perform mount point and UID / GID mapping

nano /etc/pve/lxc/108.conf

arch: amd64
cores: 2
hostname: web-debian-test
memory: 2048
mp0: /datapool/nextcloud,mp=/var/www/nextcloud-data
net0: name=eth0,bridge=vmbr1,gw=192.168.1.1,hwaddr=26:40:D5:E4:61:86,ip=192.168.1.108/24,type=veth
ostype: debian
rootfs: local-zfs:subvol-108-disk-0,size=16G
swap: 2048
unprivileged: 1
lxc.idmap: u 0 100000 33
lxc.idmap: g 0 100000 33
lxc.idmap: u 33 33 1
lxc.idmap: g 33 33 1
lxc.idmap: u 34 100034 65502
lxc.idmap: g 34 100034 65502

nano /etc/subuid

root:100000:65536
root:33:1

nano /etc/subgid

root:100000:65536
root:33:1

rebooting lxc
I will now check the rights and owners in LXC.
Inside LXC

$ ls -la /var/www
total 18
drwxr-xr-x  4 root     root      4 Jun  7 15:51 .
drwxr-xr-x 12 root     root     14 Jun  7 14:26 ..
drwxr-xr-x  4 root     root      5 Jun  7 15:23 html
drwxrwx---  2 www-data www-data  2 Jul  2 15:21 nextcloud-data

I don’t know why root web directories are nobody nogroup

$ ls -la /var/www/html
total 35
drwxr-xr-x  4 root   root        5 Jun  7 15:23 .
drwxr-xr-x  4 root   root        4 Jun  7 15:51 ..
-rw-r--r--  1 root   root    10701 Jun  7 14:26 index.html
drwxr-xr-x 13 nobody nogroup    31 Jun  7 15:02 phpmyadmin
drwxr-xr-x 14 nobody nogroup    28 Jul  3 01:17 www.nextcloud.ddns.info

When I set the chown on the host

chown 33:33 -R /rpool/data/subvol-108-disk-1/var/www

$ sudo ls -la /var/www/html         
total 35
drwxr-xr-x  4 www-data www-data     5 Jun  7 15:23 .
drwxr-xr-x  4 www-data www-data     4 Jun  7 15:51 ..
-rw-r--r--  1 www-data www-data 10701 Jun  7 14:26 index.html
drwxr-xr-x 13 www-data www-data    31 Jun  7 15:02 phpmyadmin
drwxr-xr-x 14 www-data www-data    28 Jul  3 01:17 www.nextcloud.ddns.info

However, it is very strange that the directory intended for data is empty.

sudo ls -la /var/www/nextcloud-data
total 1
drwxrwx--- 2 www-data www-data 2 Jul  2 15:21 .
drwxr-xr-x 4 root     root     4 Jun  7 15:51 ..

On host Proxmox

It is also interesting that I see the data on the host

ls -la /rpool/data/subvol-108-disk-1/var/www/nextcloud-data
total 28
drwxrwx---  4 100033 100033   8 Jun  7 15:55 .
drwxr-xr-x  4 100000 100000   4 Jun  7 15:51 ..
drwxr-xr-x 10 100033 100033  10 Jul  2 15:21 appdata_oc65vcb3bp4d
-rw-r--r--  1 100033 100033 542 Jun  7 15:54 .htaccess
-rw-r--r--  1 100033 100033   0 Jun  7 15:54 index.html
drwxr-xr-x  4 100033 100033   4 Jun  7 19:27 nextcloud
-rw-r-----  1 100033 100033 391 Jun  7 17:30 nextcloud.log
-rw-r--r--  1 100033 100033   0 Jun  7 15:54 .ocdata

I’ve been in trouble for days

stgraber · July 8, 2021, 4:07pm

Your idmap seems to be working fine. You had to do that chown because changing the map doesn’t change the existing data on disk, nobody:nogroup showed up because the owning uid/gid wasn’t part of the container’s namespace.

Now for the other part, it seems like that mount is just completely missing.
Do you see it listed in /proc/mounts inside the container? If not, can you get to the actual LXC config file that proxmox generates for this?

gusto · July 8, 2021, 6:34pm

Thank you very much for the quick reply.
Now I’ve done a LXC restore from the backup and I’m trying this procedure again. When I recursively change owner and group (33) on The host (Proxmox)

chown 33:33 -R /rpool/data/subvol-108-disk-1/var/www/nextcloud-data

So inside the LXC is still

$ ls -la /var/www
total 18
drwxr-xr-x  4 root   root     4 Jun  7 15:51 .
drwxr-xr-x 12 root   root    14 Jun  7 14:26 ..
drwxr-xr-x  4 root   root     5 Jun  7 15:23 html
drwxr-xr-x  2 nobody nogroup  2 Jul  7 18:50 nextcloud-data

I also add /proc/mounts

rpool/data/subvol-108-disk-1 / zfs rw,noatime,xattr,posixacl 0 0
datapool/nextcloud /var/www/nextcloud-data zfs rw,xattr,noacl 0 0
none /dev tmpfs rw,relatime,size=492k,mode=755,uid=100000,gid=100000 0 0
proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0
proc /proc/sys/net proc rw,nosuid,nodev,noexec,relatime 0 0
proc /proc/sys proc ro,nosuid,nodev,noexec,relatime 0 0
proc /proc/sysrq-trigger proc ro,nosuid,nodev,noexec,relatime 0 0
sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0
sysfs /sys sysfs ro,nosuid,nodev,noexec,relatime 0 0
sysfs /sys/devices/virtual/net sysfs rw,relatime 0 0
sysfs /sys/devices/virtual/net sysfs rw,nosuid,nodev,noexec,relatime 0 0
fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0
lxcfs /proc/cpuinfo fuse.lxcfs rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other 0 0
lxcfs /proc/diskstats fuse.lxcfs rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other 0 0
lxcfs /proc/loadavg fuse.lxcfs rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other 0 0
lxcfs /proc/meminfo fuse.lxcfs rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other 0 0
lxcfs /proc/stat fuse.lxcfs rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other 0 0
lxcfs /proc/swaps fuse.lxcfs rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other 0 0
lxcfs /proc/uptime fuse.lxcfs rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other 0 0
lxcfs /sys/devices/system/cpu/online fuse.lxcfs rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other 0 0
udev /dev/full devtmpfs rw,nosuid,relatime,size=16325160k,nr_inodes=4081290,mode=755 0 0
udev /dev/null devtmpfs rw,nosuid,relatime,size=16325160k,nr_inodes=4081290,mode=755 0 0
udev /dev/random devtmpfs rw,nosuid,relatime,size=16325160k,nr_inodes=4081290,mode=755 0 0
udev /dev/tty devtmpfs rw,nosuid,relatime,size=16325160k,nr_inodes=4081290,mode=755 0 0
udev /dev/urandom devtmpfs rw,nosuid,relatime,size=16325160k,nr_inodes=4081290,mode=755 0 0
udev /dev/zero devtmpfs rw,nosuid,relatime,size=16325160k,nr_inodes=4081290,mode=755 0 0
devpts /dev/console devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0
none /proc/sys/kernel/random/boot_id tmpfs ro,nosuid,nodev,noexec,relatime,size=492k,mode=755,uid=100000,gid=100000 0 0
devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=100005,mode=620,ptmxmode=666,max=1024 0 0
devpts /dev/ptmx devpts rw,nosuid,noexec,relatime,gid=100005,mode=620,ptmxmode=666,max=1024 0 0
devpts /dev/tty1 devpts rw,nosuid,noexec,relatime,gid=100005,mode=620,ptmxmode=666,max=1024 0 0
devpts /dev/tty2 devpts rw,nosuid,noexec,relatime,gid=100005,mode=620,ptmxmode=666,max=1024 0 0
tmpfs /dev/shm tmpfs rw,nosuid,nodev,uid=100000,gid=100000 0 0
tmpfs /run tmpfs rw,nosuid,nodev,mode=755,uid=100000,gid=100000 0 0
tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k,uid=100000,gid=100000 0 0
tmpfs /sys/fs/cgroup tmpfs ro,nosuid,nodev,noexec,mode=755,uid=100000,gid=100000 0 0
cgroup2 /sys/fs/cgroup/unified cgroup2 rw,nosuid,nodev,noexec,relatime 0 0
cgroup /sys/fs/cgroup/systemd cgroup rw,nosuid,nodev,noexec,relatime,xattr,name=systemd 0 0
cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0
cgroup /sys/fs/cgroup/rdma cgroup rw,nosuid,nodev,noexec,relatime,rdma 0 0
cgroup /sys/fs/cgroup/cpuset cgroup rw,nosuid,nodev,noexec,relatime,cpuset 0 0
cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0
cgroup /sys/fs/cgroup/pids cgroup rw,nosuid,nodev,noexec,relatime,pids 0 0
cgroup /sys/fs/cgroup/blkio cgroup rw,nosuid,nodev,noexec,relatime,blkio 0 0
cgroup /sys/fs/cgroup/memory cgroup rw,nosuid,nodev,noexec,relatime,memory 0 0
cgroup /sys/fs/cgroup/freezer cgroup rw,nosuid,nodev,noexec,relatime,freezer 0 0
cgroup /sys/fs/cgroup/perf_event cgroup rw,nosuid,nodev,noexec,relatime,perf_event 0 0
cgroup /sys/fs/cgroup/hugetlb cgroup rw,nosuid,nodev,noexec,relatime,hugetlb 0 0
cgroup /sys/fs/cgroup/devices cgroup rw,nosuid,nodev,noexec,relatime,devices 0 0
mqueue /dev/mqueue mqueue rw,relatime 0 0
tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3270572k,mode=700,uid=101000,gid=101000 0 0

best regards

gusto · July 10, 2021, 8:23am

lxc configuration

arch: amd64
cores: 2
hostname: web-debian-test
memory: 2048
mp0: /datapool/nextcloud,mp=/var/www/nextcloud-data
net0: name=eth0,bridge=vmbr1,gw=192.168.1.1,hwaddr=26:40:D5:E4:61:86,ip=192.168.1.108/24,type=veth
ostype: debian
rootfs: local-zfs:subvol-108-disk-0,size=16G
swap: 2048
unprivileged: 1
lxc.idmap: u 0 100000 33
lxc.idmap: g 0 100000 33
lxc.idmap: u 33 33 1
lxc.idmap: g 33 33 1
lxc.idmap: u 34 100034 65502
lxc.idmap: g 34 100034 65502

gusto · July 12, 2021, 10:19am

I still can’t move on. I still can’t change of the owner inside container from nobody:nogroup to www-data:www-data

stgraber · July 12, 2021, 12:28pm

You need to do that from outside the container.
In most cases that nobody:nogroup indicates a uid/gid outside of what’s mapped into the container so root in the container can’t change it, you need real root outside the container to do it

gusto · July 12, 2021, 2:40pm

Thank you very much for your answer.
Of course I do it on the host OS (proxmox)

root@local-proxmox:~# chown 33:33 -R /rpool/data/subvol-108-disk-1/var/www/nextcloud-data

but inside the container is always nobody nogroup

$ ls -la /var/www/    
total 18
drwxr-xr-x  4 www-data www-data  4 Jun  7 15:51 .
drwxr-xr-x 12 root     root     14 Jun  7 14:26 ..
drwxr-xr-x  4 www-data www-data  5 Jun  7 15:23 html
drwxr-xr-x  2 nobody   nogroup   2 Jul 10 09:09 nextcloud-data

I can’t restart the host OS (Proxmox). Is it possible that after restarting proxmox it could work?

stgraber · July 12, 2021, 3:39pm

That’d be because uid 33:33 isn’t passed directly into the container, so you need to shift the host value to match what it is for that container.

What’s the content of cat /proc/self/uid_map inside the container?

gusto · July 12, 2021, 4:18pm

$ cat /proc/self/uid_map
        0     100000         33
       33         33          1
       34     100034      65502