Mount disk: permission denied (ID shifting)

I’m unable to write anything to /var/cache/distfiles directory.

gentoo1 ~ # touch /var/cache/distfiles/abc
touch: cannot touch '/var/cache/distfiles/abc': Permission denied

My container config:

### This is a YAML representation of the configuration.
### Any line starting with a '# will be ignored.
###
### A sample configuration looks like:
### name: instance1
### profiles:
### - default
### config:
###   volatile.eth0.hwaddr: 00:16:3e:e9:f8:7f
### devices:
###   homedir:
###     path: /extra
###     source: /home/user
###     type: disk
### ephemeral: false
###
### Note that the name is shown but cannot be changed

architecture: x86_64
config:
  image.architecture: amd64
  image.description: Gentoo current amd64 (20210821_16:07)
  image.os: Gentoo
  image.release: current
  image.serial: "20210821_16:07"
  image.type: squashfs
  image.variant: default
  volatile.base_image: df34d189eb4eb7224156b35f05c461afef5ac550eb3cc70ce2618dd7740a27ef
  volatile.eth0.host_name: veth3e12b0b5
  volatile.eth0.hwaddr: 00:16:3e:4d:df:70
  volatile.idmap.base: "0"
  volatile.idmap.current: '[{"Isuid":true,"Isgid":false,"Hostid":1000000,"Nsid":0,"Maprange":1000000000},{"Isuid":false,"Isgid":true,"Hostid":1000000,"Nsid":0,"Maprange":1000000000}]'
  volatile.idmap.next: '[{"Isuid":true,"Isgid":false,"Hostid":1000000,"Nsid":0,"Maprange":1000000000},{"Isuid":false,"Isgid":true,"Hostid":1000000,"Nsid":0,"Maprange":1000000000}]'
  volatile.last_state.idmap: '[{"Isuid":true,"Isgid":false,"Hostid":1000000,"Nsid":0,"Maprange":1000000000},{"Isuid":false,"Isgid":true,"Hostid":1000000,"Nsid":0,"Maprange":1000000000}]'
  volatile.last_state.power: RUNNING
  volatile.uuid: 2dde94a6-4c75-42d7-a0c9-c3f5e8588738
devices:
  distfiles:
    path: /var/cache/distfiles/
    source: /var/cache/distfiles/
    type: disk
ephemeral: false
profiles:
- default
stateful: false
description: ""

Directory /var/cache/distfiles is owned by portage user and group on the host.
Any ideas how to get r/w access to that directory inside the LXD container?

Your container is running in unprivileged mode (this is a good), and its root user has been mapped to UID 1000000 on the host. Each subsequent ID >0 inside the container will be mapped to a UID of 1000000+UID inside container on the host.

As such when you share a directory as a disk device from the host into the container, the files have a UID from the host that is not shifted to the container’s UID range, and so they will not be writeable (even by root inside the container) as the UID is likely <1000000.

You can fix this by changing the ownership of the files in the shared directory to be within the container’s UID map range, or if your kernel supports it try enabling one of the UID shifting features LXD has by setting shift=true on the instance’s disk device.

See Instances | LXD

We support shiftfs and idmapped mounts for UID shifting, see LXD 4.16 has been released - #12 by toby63

I have distfiles directory mounted at rpool/ROOT/distfiles. Will this method work on top ZFS?
What do you think about adding portage user, group to /etc/subuid and /etc/subgid? /var/cache/distfiles is owned by portage:portage.

shiftfs works for ZFS (iirc).
But idmapped mounts do not work with ZFS yet.
See: LXD 4.16 has been released - #16 by stgraber

shiftfs and idmapped mounts are the best ways to gain rw-access to a host folder.
Search in the forum for more info, about how to set it up etc.

I have problem with starting container when I set shift to true inside the devices section:

$ lxc start gentoo1
Error: Failed preparing container for start: Failed to setup device mount 'distfiles': idmapping abilities are required but aren't supported on system
Try `lxc info --show-log gentoo1` for more info
$ lxc info --show-log gentoo1
Name: gentoo1
Location: none
Remote: unix://
Architecture: x86_64
Created: 2021/08/27 06:45 UTC
Status: Stopped
Type: container
Profiles: default

Log:
$ lxc config show gentoo1                                                                                                                                              
architecture: x86_64
config:
  image.architecture: amd64
  image.description: Gentoo current amd64 (20210821_16:07)
  image.os: Gentoo
  image.release: current
  image.serial: "20210821_16:07"
  image.type: squashfs
  image.variant: default
  security.syscalls.intercept.mount.shift: "true"
  volatile.base_image: df34d189eb4eb7224156b35f05c461afef5ac550eb3cc70ce2618dd7740a27ef
  volatile.eth0.hwaddr: 00:16:3e:4d:df:70
  volatile.idmap.base: "0"
  volatile.idmap.current: '[{"Isuid":true,"Isgid":false,"Hostid":1000000,"Nsid":0,"Maprange":1000000000},{"Isuid":false,"Isgid":true,"Hostid":1000000,"Nsid":0,"Maprange":1000000000}]'
  volatile.idmap.next: '[{"Isuid":true,"Isgid":false,"Hostid":1000000,"Nsid":0,"Maprange":1000000000},{"Isuid":false,"Isgid":true,"Hostid":1000000,"Nsid":0,"Maprange":1000000000}]'
  volatile.last_state.idmap: '[{"Isuid":true,"Isgid":false,"Hostid":1000000,"Nsid":0,"Maprange":1000000000},{"Isuid":false,"Isgid":true,"Hostid":1000000,"Nsid":0,"Maprange":1000000000}]'
  volatile.last_state.power: STOPPED
  volatile.uuid: 2dde94a6-4c75-42d7-a0c9-c3f5e8588738
devices:
  distfiles:
    path: /var/cache/distfiles/
    shift: "true"
    source: /var/cache/distfiles/
    type: disk
ephemeral: false
profiles:
- default
stateful: false
description: ""

You either don’t have a recent enough kernel that has idmapped mounts (or they are not supported on the backing filesystem you are using) or you don’t have the shiftfs kernel module loaded, therefore UID shifting isn’t possible on your system.

The Ubuntu kernels come with shiftfs support built in.

See

Which kernel option is responsible for enabling shiftfs?
My kernel version: 5.10.60

shiftfs is only include in the Ubuntu kernel or using @toby63’s GitHub - toby63/shiftfs-dkms: shiftfs kernel module via dkms

Starting with 5.12, there is the replacement of shiftfs though it only supports a very limited number of filesystems so far (ext4, xfs and vfat)

Can you give a link to the kernel patch that will allow enabling shifts?

It is much easier to use the dkms method, as that does not require to rebuild the kernel.

But you can find the link to the patch in my repo as well.