ZFS storage, all new containers are not remapped - why?

Incus
1

I have migrated from LXD and everything existing works as it should.

But when I launch a new container, I can see that it is not remapped?

Old container mounted on host:

ls -al /var/lib/incus/storage-pools/slower/containers/xxx/rootfs/
total 103
drwxr-xr-x  17 1000000 1000000  23 Apr 25  2022 .
d--x------   4 1000000 root      6 Apr 25  2022 ..
lrwxrwxrwx   1 1000000 1000000   7 Apr 23  2022 bin -> usr/bin
drwxr-xr-x   2 1000000 1000000   2 Apr 23  2022 boot
drwxr-xr-x   4 1000000 1000000   8 Apr 23  2022 dev
drwxr-xr-x 103 1000000 1000000 204 Jan 11 18:41 etc
drwxr-xr-x   3 1000000 1000000   3 Apr 25  2022 home
lrwxrwxrwx   1 1000000 1000000   7 Apr 23  2022 lib -> usr/lib
lrwxrwxrwx   1 1000000 1000000   9 Apr 23  2022 lib32 -> usr/lib32
lrwxrwxrwx   1 1000000 1000000   9 Apr 23  2022 lib64 -> usr/lib64
lrwxrwxrwx   1 1000000 1000000  10 Apr 23  2022 libx32 -> usr/libx32
drwxr-xr-x   2 1000000 1000000   2 Apr 23  2022 media
drwxr-xr-x   2 1000000 1000000   2 Apr 23  2022 mnt
drwxr-xr-x   2 1000000 1000000   2 Apr 23  2022 opt
drwxr-xr-x   2 1000000 1000000   2 Apr 18  2022 proc
drwx------  10 1000000 1000000  23 Jan 10 10:33 root
drwxr-xr-x   4 1000000 1000000   4 Apr 23  2022 run
lrwxrwxrwx   1 1000000 1000000   8 Apr 23  2022 sbin -> usr/sbin
drwxr-xr-x   3 1000000 1000000   3 Apr 25  2022 srv
drwxr-xr-x   2 1000000 1000000   2 Apr 18  2022 sys
drwxrwxrwt   9 1000000 1000000  11 Jul 30 20:37 tmp
drwxr-xr-x  14 1000000 1000000  14 Apr 23  2022 usr
drwxr-xr-x  12 1000000 1000000  14 Apr 25  2022 var

and new container mounted on host:

ls -al /var/lib/incus/storage-pools/slower/containers/yyy/rootfs/
total 95
drwxr-xr-x 17 root    root  23 Jan 13 14:48 .
d--x------  4 1000000 root   6 Jan 14 18:44 ..
lrwxrwxrwx  1 root    root   7 Jan 13 14:43 bin -> usr/bin
drwxr-xr-x  2 root    root   2 Apr 18  2022 boot
drwxr-xr-x  2 root    root   2 Jan 13 14:48 dev
drwxr-xr-x 69 root    root 140 Jan 14 18:49 etc
drwxr-xr-x  2 root    root   2 Apr 18  2022 home
lrwxrwxrwx  1 root    root   7 Jan 13 14:43 lib -> usr/lib
lrwxrwxrwx  1 root    root   9 Jan 13 14:43 lib32 -> usr/lib32
lrwxrwxrwx  1 root    root   9 Jan 13 14:43 lib64 -> usr/lib64
lrwxrwxrwx  1 root    root  10 Jan 13 14:43 libx32 -> usr/libx32
drwxr-xr-x  2 root    root   2 Jan 13 14:43 media
drwxr-xr-x  2 root    root   2 Jan 13 14:43 mnt
drwxr-xr-x  2 root    root   2 Jan 13 14:43 opt
drwxr-xr-x  2 root    root   2 Apr 18  2022 proc
drwx------  5 root    root  11 Jan 14 18:49 root
drwxr-xr-x  2 root    root   2 Jan 13 14:44 run
lrwxrwxrwx  1 root    root   8 Jan 13 14:43 sbin -> usr/sbin
drwxr-xr-x  2 root    root   2 Jan 13 14:43 srv
drwxr-xr-x  2 root    root   2 Apr 18  2022 sys
drwxrwxrwt  9 root    root   9 Jan 14 18:49 tmp
drwxr-xr-x 14 root    root  14 Jan 13 14:43 usr
drwxr-xr-x 12 root    root  13 Jan 13 14:43 var

Why new container is not shifted to 1000000?

I have installed incus on a new server and can see identical behaviour with default setup. I really do not understand what is going on, could you please explain why is it happening, containers seem to be privileged by default?

2

Your system must be running ZFS 2.2 which brings in support for VFS idmap.

With that in place, we no longer need to do the tedious, slow and risky shifting of the entire filesystem. Instead we can just have the kernel do it for us, so your files remain unshifted on disk but the container is still as unprivileged as before.

3

Thanks a lot, I should follow Incus development more carefully.