I also ran into the same issue. Running lxd 5.3 via snapd on an rpi. Any containers I have will fail to start if I try to mount a path that is outside of the root filesystem it seems.
Here is my config snippet from the containers:
homedir:
path: /home/david/
source: /home/david/container-home/
type: disk
storage:
path: /storage/incoming/
source: /storage/incoming/
type: disk
the homedir mounts fine, however storage will fail to start in the same exact way as the OP’s log. my /storage path is simply:
/dev/sda1 on /storage type ext4 (rw,noexec,relatime,stripe=8191)
I am wondering if it is because of noexec flag? I can probably test later on this but at the moment, I cannot take down my containers again. I’ve since reverted to 5.2 and locked to the 5.2-stable channel.
To be clear, we can clearly see that there is a problem and we’d love to fix it but the log doesn’t provide enough details to figure out what’s going on and we’ve so far been unable to reproduce it on one of our own systems.
So we either need step by step instructions to reproduce this on a clean system or need access to an affected system so we can hopefully see what’s going on and then reproduce this ourselves.
Same here. I cant mount my RCLONE mounts from my host. my zfs mount works fine.
My rclone mounts have worked fine for years. No over night they failed.
Edit: I can add the mounts if the lxc is running. But if I restart /stop/start it fails. Ive also tried adding when the lxc is down and then start. Also fails.
lxc test 20220630202354.768 WARN conf - …/src/src/lxc/conf.c:lxc_map_ids:3592 - newuidmap binary is missing
lxc test 20220630202354.768 WARN conf - …/src/src/lxc/conf.c:lxc_map_ids:3598 - newgidmap binary is missing
lxc test 20220630202354.769 WARN conf - …/src/src/lxc/conf.c:lxc_map_ids:3592 - newuidmap binary is missing
lxc test 20220630202354.769 WARN conf - …/src/src/lxc/conf.c:lxc_map_ids:3598 - newgidmap binary is missing
lxc test 20220630202354.770 WARN cgfsng - …/src/src/lxc/cgroups/cgfsng.c:fchowmodat:1252 - No such file or directory - Failed to fchownat(42, memory.oom.group, 1000000000, 0, AT_EMPTY_PATH | AT_SYMLINK_NOFOLLOW )
lxc test 20220630202354.888 ERROR conf - …/src/src/lxc/conf.c:mount_entry:2459 - Operation not permitted - Failed to mount “/var/snap/lxd/common/lxd/devices/test/disk.CBR.home-david-rclone” on “/var/snap/lxd/common/lxc//home/david/rclone”
lxc test 20220630202354.888 ERROR conf - …/src/src/lxc/conf.c:lxc_setup:4375 - Failed to setup mount entries
lxc test 20220630202354.888 ERROR start - …/src/src/lxc/start.c:do_start:1275 - Failed to setup container “test”
lxc test 20220630202354.888 ERROR sync - …/src/src/lxc/sync.c:sync_wait:34 - An error occurred in another process (expected sequence number 3)
lxc test 20220630202354.895 WARN network - …/src/src/lxc/network.c:lxc_delete_network_priv:3631 - Failed to rename interface with index 0 from “eth0” to its initial name “veth0d2b4b0a”
lxc test 20220630202354.895 ERROR lxccontainer - …/src/src/lxc/lxccontainer.c:wait_on_daemonized_start:877 - Received container state “ABORTING” instead of “RUNNING”
lxc test 20220630202354.895 ERROR start - …/src/src/lxc/start.c:__lxc_start:2074 - Failed to spawn container “test”
lxc test 20220630202354.895 WARN start - …/src/src/lxc/start.c:lxc_abort:1039 - No such process - Failed to send SIGKILL via pidfd 43 for process 1086490
lxc test 20220630202400.128 WARN conf - …/src/src/lxc/conf.c:lxc_map_ids:3592 - newuidmap binary is missing
lxc test 20220630202400.129 WARN conf - …/src/src/lxc/conf.c:lxc_map_ids:3598 - newgidmap binary is missing
lxc 20220630202400.676 ERROR af_unix - …/src/src/lxc/af_unix.c:lxc_abstract_unix_recv_fds_iov:218 - Connection reset by peer - Failed to receive response
lxc 20220630202400.676 ERROR commands - …/src/src/lxc/commands.c:lxc_cmd_rsp_recv_fds:127 - Failed to receive file descriptors for command “get_state”
I’m hitting this error as well, but only on NFS shares (mounted on the host) added as devices to the containers. Other containers that have regular folders from ext4 drives shared to them start fine.
This is on Jammy, kernel 5.15.0-40-generic. As others have done, reverting the LXD snap to 5.2 allows starting the containers with the same nfs shares without issues.
I ran into this out of the blue today as well. One observation is that config device add does not exhibit the failure while the container is running, even for the exact same disks that cause the container to fail during startup.
This works fine:
lxc start tester
lxc config device add tester test1 disk source=/data/prizm/nasfs/dvr path=/mnt/dvr
But then lxc exec tester reboot… and the instance fails to come back up. Remove the disk, instance starts. Run device add again and it works, until you restart the container.
Thanks, that suggests it is perhaps some regression in liblxc itself rather than lxd, or done difference in how the disk devices are passed at start time vs when running. This is a useful data point and I’ll try and recreate it. Thanks
