Failed to change ownership on new container storage

Hello,

In trying to create a container, I encounter a problem I never had. There seems to be a problem with the right on the disk deployed after downloading an image. The problem does not occur if I copy an existing container.

Thanks for your read and help.


Problem:

root@nyx:/mnt/ssd/containers# lxc launch images:debian/buster toto
Creating toto
Starting toto                                 
Error: Failed preparing container for start: Failed to change ownership of: /var/snap/lxd/common/lxd/storage-pools/local/containers/toto/rootfs/usr/share/i18n/charmaps/NF_Z_62-010_1973.gz
Try `lxc info --show-log local:toto` for more info
root@nyx:/mnt/ssd/containers# ls -lah | grep -E "toto|gw"
d--x------ 1 1000000 root  78 oct.   1 16:13 gw
d--x------ 1 root    root  78 févr.  8 14:36 toto
pulsar@nyx:~$ lxc launch images:debian/buster toto
Creating toto
Starting toto
Error: Failed preparing container for start: Failed to change ownership of: /var/snap/lxd/common/lxd/storage-pools/local/containers/toto/rootfs/usr/share/i18n/charmaps/NF_Z_62-010_1973.gz
pulsar@nyx:~$ sudo ls -lah /mnt/ssd/containers | grep -E "toto|gw"
d--x------ 1 1000000 root  78 oct.   1 16:13 gw
d--x------ 1 root    root  78 févr.  8 14:46 toto

Host informations:

root@nyx:/mnt/ssd/containers# cat /etc/debian_version 
10.8
root@nyx:/mnt/ssd/containers# snap list lxd
Name  Version  Rev    Tracking       Publisher   Notes
lxd   4.10     19168  latest/stable  canonical✓  -

For information, snapd does not see version 4.11 at the moment.

What kernel are you running?

Oh, sorry and thanks for your answer:

pulsar@nyx:~$ uname -a
Linux nyx 5.10.12-rockchip64 #21.02.1 SMP PREEMPT Wed Feb 3 20:55:02 CET 2021 aarch64 GNU/Linux 

Okay, I know there are some weird issues with 5.10 around things like readlink and path resolution which this particular function (shiftowner) is using quite extensively…

@brauner ideas?

Do you want me to try to downgrade the kernel?

If you can, that’d be interesting to know if that’s the issue.

I just downgraded the kernel, but:

pulsar@nyx:~$ lxc launch images:debian/buster toto
Creating toto
Starting toto
Error: Failed preparing container for start: Failed to change ownership of: /var/snap/lxd/common/lxd/storage-pools/local/containers/toto/rootfs/usr/share/i18n/charmaps/NF_Z_62-010_1973.gz
Try `lxc info --show-log local:toto` for more info
pulsar@nyx:~$ sudo ls -lah /mnt/ssd/containers | grep -E "toto|gw"
d--x------ 1 1000000 root  78 oct.   1 16:13 gw
d--x------ 1 root    root  78 févr.  8 16:06 toto
pulsar@nyx:~$ uname -a
Linux nyx 5.9.14-rockchip64 #20.11.4 SMP PREEMPT Tue Dec 15 08:52:20 CET 2020 aarch64 GNU/Linux

Thanks for checking.

Any additional errors show in journalctl -u snap.lxd.daemon -n 30?

Indeed, I hadn’t thought about it, here it is:

pulsar@nyx:~$ journalctl -u snap.lxd.daemon -n 30
-- Logs begin at Mon 2021-02-08 16:02:36 CET, end at Mon 2021-02-08 16:28:54 CET. --
févr. 08 16:03:19 nyx lxd.daemon[1240]:   7: fd:  12: net_cls,net_prio
févr. 08 16:03:19 nyx lxd.daemon[1240]:   8: fd:  13: pids
févr. 08 16:03:19 nyx lxd.daemon[1240]:   9: fd:  14: memory
févr. 08 16:03:19 nyx lxd.daemon[1240]:  10: fd:  15: rdma
févr. 08 16:03:19 nyx lxd.daemon[1240]:  11: fd:  16: freezer
févr. 08 16:03:19 nyx lxd.daemon[1240]:  12: fd:  17: blkio
févr. 08 16:03:19 nyx lxd.daemon[1240]: Kernel supports pidfds
févr. 08 16:03:19 nyx lxd.daemon[1240]: Kernel supports swap accounting
févr. 08 16:03:19 nyx lxd.daemon[1240]: api_extensions:
févr. 08 16:03:19 nyx lxd.daemon[1240]: - cgroups
févr. 08 16:03:19 nyx lxd.daemon[1240]: - sys_cpu_online
févr. 08 16:03:19 nyx lxd.daemon[1240]: - proc_cpuinfo
févr. 08 16:03:19 nyx lxd.daemon[1240]: - proc_diskstats
févr. 08 16:03:19 nyx lxd.daemon[1240]: - proc_loadavg
févr. 08 16:03:19 nyx lxd.daemon[1240]: - proc_meminfo
févr. 08 16:03:19 nyx lxd.daemon[1240]: - proc_stat
févr. 08 16:03:19 nyx lxd.daemon[1240]: - proc_swaps
févr. 08 16:03:19 nyx lxd.daemon[1240]: - proc_uptime
févr. 08 16:03:19 nyx lxd.daemon[1240]: - shared_pidns
févr. 08 16:03:19 nyx lxd.daemon[1240]: - cpuview_daemon
févr. 08 16:03:19 nyx lxd.daemon[1240]: - loadavg_daemon
févr. 08 16:03:19 nyx lxd.daemon[1240]: - pidfds
févr. 08 16:03:20 nyx lxd.daemon[1240]: => Starting LXD
févr. 08 16:03:20 nyx lxd.daemon[1240]: t=2021-02-08T16:03:20+0100 lvl=warn msg=" - Couldn't find the CGroup blkio.weight, disk priority will be ignored"
févr. 08 16:03:44 nyx lxd.daemon[1240]: 2021/02/08 16:03:44 http: superfluous response.WriteHeader call from github.com/lxc/lxd/lxd/response.(*errorResponse).Render (response.go:224)
févr. 08 16:04:20 nyx lxd.daemon[1240]: => LXD is ready
févr. 08 16:05:42 nyx lxd.daemon[1240]: Failed chown: Disk quota exceeded
févr. 08 16:05:47 nyx lxd.daemon[1240]: 2021/02/08 16:05:47 http: superfluous response.WriteHeader call from github.com/lxc/lxd/lxd/response.(*errorResponse).Render (response.go:224)
févr. 08 16:06:45 nyx lxd.daemon[1240]: Failed chown: Disk quota exceeded
févr. 08 16:28:54 nyx lxd.daemon[1240]: Failed chown: Disk quota exceeded

In addition:

pulsar@nyx:~$ df -h /mnt/ssd
Sys. de fichiers Taille Utilisé Dispo Uti% Monté sur
/dev/sda           112G     27G   85G  25% /mnt/ssd
pulsar@nyx:~$ lxc storage show local | head -n7
config:
  size: 15GB
  source: /mnt/ssd
  volatile.initial_source: /mnt/ssd
description: ""
name: local
driver: btrfs
pulsar@nyx:~$ lxc storage info local | head -n 6
info:
  description: ""
  driver: btrfs
  name: local
  space used: 28.70GB
  total space: 120.03GB

I just found the solution. In the default profile, I declare that rootfs is 1GB. This has never been a problem so far, but changing to 8GB (2GB is not enough) that works fine.

I think this is a regression. What do you think about it?

Before:

pulsar@nyx:~$ lxc profile show default | head -n17
config:
[...]
  root:
    path: /
    pool: local
    size: 1GB
    type: disk
[...]

After:

pulsar@nyx:~$ lxc profile show default | head -n17
config:
[...]
  root:
    path: /
    pool: local
    size: 8GB
    type: disk
[...]

Result:

pulsar@nyx:~$ lxc info toto
[...]
  Disk usage:
    root: 16.38kB
[...]

We haven’t changed anything in that part of the code in a long time, but it indeed does look like you hit some kind of kernel quota here during remap.

btrfs is particularly weird with qgroup quotas…

Okay, I see, thank you.
I plan to move to Ceph in a few weeks by adding two odroid c4 to my rock64. I think that will solve part of the problem. Hopefully Gigabit won’t kill performance too much.

Solved! Thanks again!