I have hit by a problem where I cannot allocate/initiate any containers. My lxd is using zfs dataset: fast/lxd. This stirage pool has more than 900GB available capacity but after stating 70 instances it reports an error for each of those failed containers:
lxc info --show-log juju-5cb34f-75
Name: juju-5cb34f-75
Status: STOPPED
Type: container
Architecture: x86_64
Location: testbed
Created: 2022/07/25 17:51 UTC
Last Used: 2022/07/25 17:52 UTC
Log:
lxc juju-5cb34f-75 20220725175203.848 WARN conf - ../src/src/lxc/conf.c:lxc_map_ids:3592 - newuidmap binary is missing
lxc juju-5cb34f-75 20220725175203.848 WARN conf - ../src/src/lxc/conf.c:lxc_map_ids:3598 - newgidmap binary is missing
lxc juju-5cb34f-75 20220725175203.854 WARN conf - ../src/src/lxc/conf.c:lxc_map_ids:3592 - newuidmap binary is missing
lxc juju-5cb34f-75 20220725175203.854 WARN conf - ../src/src/lxc/conf.c:lxc_map_ids:3598 - newgidmap binary is missing
lxc juju-5cb34f-75 20220725175203.976 WARN conf - ../src/src/lxc/conf.c:lxc_setup_keyring:4074 - Disk quota exceeded - Failed to create kernel keyring
lxc juju-5cb34f-75 20220725175206.882 WARN conf - ../src/src/lxc/conf.c:lxc_map_ids:3592 - newuidmap binary is missing
lxc juju-5cb34f-75 20220725175206.882 WARN conf - ../src/src/lxc/conf.c:lxc_map_ids:3598 - newgidmap binary is missing
lxc 20220725175206.950 ERROR af_unix - ../src/src/lxc/af_unix.c:lxc_abstract_unix_recv_fds_iov:218 - Connection reset by peer - Failed to receive response
lxc 20220725175206.950 ERROR commands - ../src/src/lxc/commands.c:lxc_cmd_rsp_recv_fds:128 - Failed to receive file descriptors for command "get_state"```
zfs history shows this:
`2022-07-25.17:51:51 zfs clone fast/lxd/images/e9589b6e9c886888b3df98aee0f0e16c5805383418b3563cd8845220f43b40ff@readonly fast/lxd/containers/juju-5cb34f-75
2022-07-25.17:51:51 zfs set mountpoint=legacy canmount=noauto fast/lxd/containers/juju-5cb34f-75
2022-07-25.17:51:51 zfs set mountpoint=legacy canmount=noauto fast/lxd/containers/juju-5cb34f-75
2022-07-25.17:51:51 zfs set quota=none fast/lxd/containers/juju-5cb34f-75
2022-07-25.17:51:51 zfs set refquota=none fast/lxd/containers/juju-5cb34f-75
2022-07-25.17:51:51 zfs set reservation=none fast/lxd/containers/juju-5cb34f-75
2022-07-25.17:51:51 zfs set refreservation=none fast/lxd/containers/juju-5cb34f-75
2022-07-25.17:51:52 zfs set quota=none fast/lxd/images/e9589b6e9c886888b3df98aee0f0e16c5805383418b3563cd8845220f43b40ff
2022-07-25.17:51:52 zfs set refquota=none fast/lxd/images/e9589b6e9c886888b3df98aee0f0e16c5805383418b3563cd8845220f43b40ff
2022-07-25.17:51:52 zfs set reservation=none fast/lxd/images/e9589b6e9c886888b3df98aee0f0e16c5805383418b3563cd8845220f43b40ff
2022-07-25.17:51:52 zfs set refreservation=none fast/lxd/images/e9589b6e9c886888b3df98aee0f0e16c5805383418b3563cd8845220f43b40ff
2022-07-25.17:51:52 zfs set mountpoint=legacy canmount=noauto fast/lxd/containers/juju-5cb34f-75
2022-07-25.17:51:53 zfs clone fast/lxd/images/e9589b6e9c886888b3df98aee0f0e16c5805383418b3563cd8845220f43b40ff@readonly fast/lxd/containers/juju-5cb34f-76
2022-07-25.17:51:53 zfs set quota=none fast/lxd/images/e9589b6e9c886888b3df98aee0f0e16c5805383418b3563cd8845220f43b40ff
2022-07-25.17:51:53 zfs set mountpoint=legacy canmount=noauto fast/lxd/containers/juju-5cb34f-76
2022-07-25.17:51:53 zfs set refquota=none fast/lxd/images/e9589b6e9c886888b3df98aee0f0e16c5805383418b3563cd8845220f43b40ff
2022-07-25.17:51:53 zfs set reservation=none fast/lxd/images/e9589b6e9c886888b3df98aee0f0e16c5805383418b3563cd8845220f43b40ff
2022-07-25.17:51:53 zfs set mountpoint=legacy canmount=noauto fast/lxd/containers/juju-5cb34f-76
2022-07-25.17:51:53 zfs set refreservation=none fast/lxd/images/e9589b6e9c886888b3df98aee0f0e16c5805383418b3563cd8845220f43b40ff
2022-07-25.17:51:53 zfs set mountpoint=legacy canmount=noauto fast/lxd/containers/juju-5cb34f-75
2022-07-25.17:51:53 zfs set quota=none fast/lxd/containers/juju-5cb34f-76
2022-07-25.17:51:53 zfs clone fast/lxd/images/e9589b6e9c886888b3df98aee0f0e16c5805383418b3563cd8845220f43b40ff@readonly fast/lxd/containers/juju-5cb34f-77
2022-07-25.17:51:53 zfs set quota=none fast/lxd/images/e9589b6e9c886888b3df98aee0f0e16c5805383418b3563cd8845220f43b40ff
2022-07-25.17:51:53 zfs set refquota=none fast/lxd/containers/juju-5cb34f-76
2022-07-25.17:51:53 zfs set refquota=none fast/lxd/images/e9589b6e9c886888b3df98aee0f0e16c5805383418b3563cd8845220f43b40ff
2022-07-25.17:51:53 zfs set reservation=none fast/lxd/containers/juju-5cb34f-76
2022-07-25.17:51:53 zfs set mountpoint=legacy canmount=noauto fast/lxd/containers/juju-5cb34f-77
2022-07-25.17:51:53 zfs set reservation=none fast/lxd/images/e9589b6e9c886888b3df98aee0f0e16c5805383418b3563cd8845220f43b40ff
2022-07-25.17:51:53 zfs set refreservation=none fast/lxd/images/e9589b6e9c886888b3df98aee0f0e16c5805383418b3563cd8845220f43b40ff
2022-07-25.17:51:53 zfs set refreservation=none fast/lxd/containers/juju-5cb34f-76
2022-07-25.17:51:53 zfs set mountpoint=legacy canmount=noauto fast/lxd/containers/juju-5cb34f-77
2022-07-25.17:51:53 zfs clone fast/lxd/images/e9589b6e9c886888b3df98aee0f0e16c5805383418b3563cd8845220f43b40ff@readonly fast/lxd/containers/juju-5cb34f-78
2022-07-25.17:51:53 zfs set mountpoint=legacy canmount=noauto fast/lxd/containers/juju-5cb34f-78
2022-07-25.17:51:53 zfs set quota=none fast/lxd/containers/juju-5cb34f-77
2022-07-25.17:51:53 zfs set mountpoint=legacy canmount=noauto fast/lxd/containers/juju-5cb34f-78
2022-07-25.17:51:53 zfs set refquota=none fast/lxd/containers/juju-5cb34f-77
2022-07-25.17:51:53 zfs set mountpoint=legacy canmount=noauto fast/lxd/containers/juju-5cb34f-76
2022-07-25.17:51:53 zfs set reservation=none fast/lxd/containers/juju-5cb34f-77
2022-07-25.17:51:53 zfs set refreservation=none fast/lxd/containers/juju-5cb34f-77
2022-07-25.17:51:53 zfs set quota=none fast/lxd/containers/juju-5cb34f-78
2022-07-25.17:51:53 zfs set refquota=none fast/lxd/containers/juju-5cb34f-78
2022-07-25.17:51:53 zfs set reservation=none fast/lxd/containers/juju-5cb34f-78
2022-07-25.17:51:53 zfs set refreservation=none fast/lxd/containers/juju-5cb34f-78
2022-07-25.17:51:53 zfs set mountpoint=legacy canmount=noauto fast/lxd/containers/juju-5cb34f-76
2022-07-25.17:51:53 zfs set mountpoint=legacy canmount=noauto fast/lxd/containers/juju-5cb34f-77
2022-07-25.17:51:53 zfs set mountpoint=legacy canmount=noauto fast/lxd/containers/juju-5cb34f-78
2022-07-25.17:51:54 zfs set mountpoint=legacy canmount=noauto fast/lxd/containers/juju-5cb34f-77
2022-07-25.17:51:54 zfs set mountpoint=legacy canmount=noauto fast/lxd/containers/juju-5cb34f-78`
These last 4 containers cannot start or any other container that I am trying to start gives the same error.