NFS devices added as a directory in the container: "chown: changing ownership of 'testfile': Operation not permitted"

I have a NFS (Amazon EFS) share mounted on Ubuntu 20.04 LXD server using the following /etc/fstab entry on the host:

10.0.0.210:/ /storage nfs4 nfsvers=4.1,rsize=1048576,wsize=1048576,hard,timeo=600,retrans=2,noresvport 0 0

I’ve verified on the host that I’m able to create files and directories as root, and chown them to other users.

Now, I’ve created a /storage/containers directory inside the NFS mount, and added it to the container as a device using:

lxc config device add testcontainer storage disk source=/storage/containers path=/storage

I’ve also made “chown 1000000:1000000 /storage/containers” to make sure the UIDs are matching the container.

With this in place, I’m able to add files and directories in the container, as root. However, trying to use chown in the container renders an error:

# touch testfile
# chown www-data:www-data testfile 
chown: changing ownership of 'testfile': Operation not permitted

Why is this happening? If I add a normal, non-NFS directory as a container device, I’m able to use chown in the container. Also, I’m able to use chown on host.
Any help appreciated!

Feels like an NFS issue. NFS has its own concept of uid/gid mapping and of who is allowed to change uids/gids for existing files.

This is most likely what’s causing issues here. If I had to guess, NFS is seeing root in your container (uid 1000000) as trying to change ownership and refusing because it’s not real root (uid 0).

NFSv3 is a lot less smart about that, so if supported, it may behave better in this case.
Otherwise, you may want to look at mount options and see if there’s something in there to change this behavior.

Nope, no luck with NFSv3 either.

What you’re describing (non-root trying to chown) is probably what’s happening from NFS point of view, and hence the error. Too bad.