Hi all.
Ubuntu 20.04
LXD version: 5.3
This morning when my server rebooted, suddenly none of the containers would start up.
Turns out it’s because there is an attached folder (“device”? “disk”?) that errors with -
Operation not permitted - Failed to mount
Full log:
root@api:~# lxc info --show-log OutBoundAPI-dev
Name: OutBoundAPI-dev
Status: STOPPED
Type: container
Architecture: x86_64
Created: 2022/03/15 17:29 UTC
Last Used: 2022/07/03 14:02 UTC
Log:
lxc OutBoundAPI-dev 20220703140242.500 WARN conf - ../src/src/lxc/conf.c:lxc_map_ids:3592 - newuidmap binary is missing
lxc OutBoundAPI-dev 20220703140242.500 WARN conf - ../src/src/lxc/conf.c:lxc_map_ids:3598 - newgidmap binary is missing
lxc OutBoundAPI-dev 20220703140242.510 WARN conf - ../src/src/lxc/conf.c:lxc_map_ids:3592 - newuidmap binary is missing
lxc OutBoundAPI-dev 20220703140242.511 WARN conf - ../src/src/lxc/conf.c:lxc_map_ids:3598 - newgidmap binary is missing
lxc OutBoundAPI-dev 20220703140242.520 WARN cgfsng - ../src/src/lxc/cgroups/cgfsng.c:fchowmodat:1252 - No such file or directory - Failed to fchownat(40, memory.oom.group, 1000000000, 0, AT_EMPTY_PATH | AT_SYMLINK_NOFOLLOW )
lxc OutBoundAPI-dev 20220703140242.129 ERROR conf - ../src/src/lxc/conf.c:mount_entry:2459 - Operation not permitted - Failed to mount "/var/snap/lxd/common/lxd/devices/OutBoundAPI-dev/disk.uploads.uploads-" on "/var/snap/lxd/common/lxc//uploads/"
lxc OutBoundAPI-dev 20220703140242.129 ERROR conf - ../src/src/lxc/conf.c:lxc_setup:4375 - Failed to setup mount entries
lxc OutBoundAPI-dev 20220703140242.129 ERROR start - ../src/src/lxc/start.c:do_start:1275 - Failed to setup container "OutBoundAPI-dev"
lxc OutBoundAPI-dev 20220703140242.129 ERROR sync - ../src/src/lxc/sync.c:sync_wait:34 - An error occurred in another process (expected sequence number 3)
lxc OutBoundAPI-dev 20220703140242.136 WARN network - ../src/src/lxc/network.c:lxc_delete_network_priv:3631 - Failed to rename interface with index 0 from "eth0" to its initial name "veth86485039"
lxc OutBoundAPI-dev 20220703140242.136 ERROR lxccontainer - ../src/src/lxc/lxccontainer.c:wait_on_daemonized_start:877 - Received container state "ABORTING" instead of "RUNNING"
lxc OutBoundAPI-dev 20220703140242.136 ERROR start - ../src/src/lxc/start.c:__lxc_start:2074 - Failed to spawn container "OutBoundAPI-dev"
lxc OutBoundAPI-dev 20220703140242.136 WARN start - ../src/src/lxc/start.c:lxc_abort:1039 - No such process - Failed to send SIGKILL via pidfd 41 for process 88112
lxc OutBoundAPI-dev 20220703140247.211 WARN conf - ../src/src/lxc/conf.c:lxc_map_ids:3592 - newuidmap binary is missing
lxc OutBoundAPI-dev 20220703140247.211 WARN conf - ../src/src/lxc/conf.c:lxc_map_ids:3598 - newgidmap binary is missing
lxc 20220703140247.233 ERROR af_unix - ../src/src/lxc/af_unix.c:lxc_abstract_unix_recv_fds_iov:218 - Connection reset by peer - Failed to receive response
lxc 20220703140247.233 ERROR commands - ../src/src/lxc/commands.c:lxc_cmd_rsp_recv_fds:127 - Failed to receive file descriptors for command "get_state"
This is a shared folder from another samba server that is mounted in the host with cifs-utils then attached to the container with: lxc config device add
lxc.conf =
lxc.mount.entry = /var/snap/lxd/common/lxd/devices/OutBoundAPI-dev/disk.uploads.uploads- uploads/ none bind,create=dir 0 0
On the host, fstab =
//192.168.0.10/uploads /mnt/uploads cifs _netdev,username=xxx,password=xxx,rw,user,noperm 0 0
I originally added the device with:
lxc config device add OutBoundAPI-dev uploads disk source=/mnt/uploads/ path=/uploads/
NOTE:
It has been this way, running flawlessly through reboots, for months. Then suddenly this morning, just … stops working.
The most confusing thing is that if I manually remove the device, then start the container, then add the device - it works. But when I reboot, it fails to start again.
Any thoughts or ideas are welcome. I am not sure what other info is pertinent to post or share?
Complete lxc.conf =
lxc.log.file = /var/snap/lxd/common/lxd/logs/OutBoundAPI-dev/lxc.log
lxc.log.level = warn
lxc.console.buffer.size = auto
lxc.console.size = auto
lxc.console.logfile = /var/snap/lxd/common/lxd/logs/OutBoundAPI-dev/console.log
lxc.mount.auto = proc:rw sys:rw cgroup:mixed
lxc.autodev = 1
lxc.pty.max = 1024
lxc.mount.entry = /dev/fuse dev/fuse none bind,create=file,optional 0 0
lxc.mount.entry = /dev/net/tun dev/net/tun none bind,create=file,optional 0 0
lxc.mount.entry = /proc/sys/fs/binfmt_misc proc/sys/fs/binfmt_misc none rbind,create=dir,optional 0 0
lxc.mount.entry = /sys/fs/fuse/connections sys/fs/fuse/connections none rbind,create=dir,optional 0 0
lxc.mount.entry = /sys/fs/pstore sys/fs/pstore none rbind,create=dir,optional 0 0
lxc.mount.entry = /sys/kernel/config sys/kernel/config none rbind,create=dir,optional 0 0
lxc.mount.entry = /sys/kernel/debug sys/kernel/debug none rbind,create=dir,optional 0 0
lxc.mount.entry = /sys/kernel/security sys/kernel/security none rbind,create=dir,optional 0 0
lxc.mount.entry = /sys/kernel/tracing sys/kernel/tracing none rbind,create=dir,optional 0 0
lxc.mount.entry = /dev/mqueue dev/mqueue none rbind,create=dir,optional 0 0
lxc.include = /snap/lxd/current/lxc/config//common.conf.d/
lxc.arch = linux64
lxc.hook.version = 1
lxc.hook.pre-start = /proc/1801/exe callhook /var/snap/lxd/common/lxd "default" "OutBoundAPI-dev" start
lxc.hook.stop = /snap/lxd/current/bin/lxd callhook /var/snap/lxd/common/lxd "default" "OutBoundAPI-dev" stopns
lxc.hook.post-stop = /snap/lxd/current/bin/lxd callhook /var/snap/lxd/common/lxd "default" "OutBoundAPI-dev" stop
lxc.tty.max = 0
lxc.uts.name = OutBoundAPI-dev
lxc.mount.entry = /var/snap/lxd/common/lxd/devlxd dev/lxd none bind,create=dir 0 0
lxc.apparmor.profile = lxd-OutBoundAPI-dev_</var/snap/lxd/common/lxd>//&:lxd-OutBoundAPI-dev_<var-snap-lxd-common-lxd>:
lxc.seccomp.profile = /var/snap/lxd/common/lxd/security/seccomp/OutBoundAPI-dev
lxc.idmap = u 0 1000000 1000000000
lxc.idmap = g 0 1000000 1000000000
lxc.mount.auto = shmounts:/var/snap/lxd/common/lxd/shmounts/OutBoundAPI-dev:/dev/.lxd-mounts
lxc.net.0.type = phys
lxc.net.0.name = eth0
lxc.net.0.flags = up
lxc.net.0.link = veth86485039
lxc.rootfs.path = dir:/var/snap/lxd/common/lxd/storage-pools/default/containers/OutBoundAPI-dev/rootfs
lxc.mount.entry = /var/snap/lxd/common/lxd/devices/OutBoundAPI-dev/disk.uploads.uploads- uploads/ none bind,create=dir 0 0
Thanks in advance for any insights.