Incus Container Wont stop or allow itself to be deleted

I have rebuilt from LXD Incus 4 server cluster, and I have one instance that is stuck. Can’t stop or remove it via Incus command line. Wondering if it is safe to manually delete it off disk or is there a better way. Everything else seems to run fine. Any ideas?

Also getting this in some new containers on their startup.
These are been brought in from LXD
root@Q3:/home/ic2000# incus info --show-log WP-SPACEWATCH2024
Name: WP-SPACEWATCH2024
Status: STOPPED
Type: container
Architecture: x86_64
Location: Q1
Created: 2024/01/01 03:00 EST
Last Used: 2024/01/01 03:01 EST

Log:

lxc WP-SPACEWATCH2024 20240101080101.541 WARN conf - …/src/lxc/conf.c:lxc_map_ids:3621 - newuidmap binary is missing
lxc WP-SPACEWATCH2024 20240101080101.565 WARN conf - …/src/lxc/conf.c:lxc_map_ids:3627 - newgidmap binary is missing
lxc WP-SPACEWATCH2024 20240101080101.566 WARN conf - …/src/lxc/conf.c:lxc_map_ids:3621 - newuidmap binary is missing
lxc WP-SPACEWATCH2024 20240101080101.566 WARN conf - …/src/lxc/conf.c:lxc_map_ids:3627 - newgidmap binary is missing
lxc WP-SPACEWATCH2024 20240101080101.566 WARN cgfsng - …/src/lxc/cgroups/cgfsng.c:fchowmodat:1619 - No such file or directory - Failed to fchownat(40, memory.oom.group, 1000000000, 0, AT_EMPTY_PATH | AT_SYMLINK_NOFOLLOW )

root@Q3:/home/ic2000# incus start WP-SPACEWATCH2024
Error: Failed to run: /opt/incus/bin/incusd forkstart WP-SPACEWATCH2024 /var/lib/incus/containers /var/log/incus/WP-SPACEWATCH2024/lxc.conf: exit status 1
Try incus info --show-log WP-SPACEWATCH2024 for more info
root@Q3:/home/ic2000# incus info --show-log WP-SPACEWATCH2024
Name: WP-SPACEWATCH2024
Status: STOPPED
Type: container
Architecture: x86_64
Location: Q1
Created: 2024/01/01 03:00 EST
Last Used: 2024/01/01 04:04 EST

Log:

lxc WP-SPACEWATCH2024 20240101090455.639 WARN conf - …/src/lxc/conf.c:lxc_map_ids:3621 - newuidmap binary is missing
lxc WP-SPACEWATCH2024 20240101090455.639 WARN conf - …/src/lxc/conf.c:lxc_map_ids:3627 - newgidmap binary is missing
lxc WP-SPACEWATCH2024 20240101090455.640 WARN conf - …/src/lxc/conf.c:lxc_map_ids:3621 - newuidmap binary is missing
lxc WP-SPACEWATCH2024 20240101090455.640 WARN conf - …/src/lxc/conf.c:lxc_map_ids:3627 - newgidmap binary is missing
lxc WP-SPACEWATCH2024 20240101090455.641 WARN cgfsng - …/src/lxc/cgroups/cgfsng.c:fchowmodat:1619 - No such file or directory - Failed to fchownat(40, memory.oom.group, 1000000000, 0, AT_EMPTY_PATH | AT_SYMLINK_NOFOLLOW )
lxc WP-SPACEWATCH2024 20240101090455.770 ERROR conf - …/src/lxc/conf.c:run_buffer:322 - Script exited with status 32
lxc WP-SPACEWATCH2024 20240101090455.770 ERROR conf - …/src/lxc/conf.c:lxc_setup:4437 - Failed to run mount hooks
lxc WP-SPACEWATCH2024 20240101090455.771 ERROR start - …/src/lxc/start.c:do_start:1272 - Failed to setup container “WP-SPACEWATCH2024”
lxc WP-SPACEWATCH2024 20240101090455.771 ERROR sync - …/src/lxc/sync.c:sync_wait:34 - An error occurred in another process (expected sequence number 4)
lxc WP-SPACEWATCH2024 20240101090455.778 WARN network - …/src/lxc/network.c:lxc_delete_network_priv:3631 - Failed to rename interface with index 0 from “eth0” to its initial name “vetha73f8f3e”
lxc WP-SPACEWATCH2024 20240101090455.778 ERROR lxccontainer - …/src/lxc/lxccontainer.c:wait_on_daemonized_start:878 - Received container state “ABORTING” instead of “RUNNING”
lxc WP-SPACEWATCH2024 20240101090455.778 ERROR start - …/src/lxc/start.c:__lxc_start:2107 - Failed to spawn container “WP-SPACEWATCH2024”
lxc WP-SPACEWATCH2024 20240101090455.778 WARN start - …/src/lxc/start.c:lxc_abort:1036 - No such process - Failed to send SIGKILL via pidfd 41 for process 27343
lxc 20240101090455.930 ERROR af_unix - …/src/lxc/af_unix.c:lxc_abstract_unix_recv_fds_iov:218 - Connection reset by peer - Failed to receive response
lxc 20240101090455.930 ERROR commands - …/src/lxc/commands.c:lxc_cmd_rsp_recv_fds:128 - Failed to receive file descriptors for command “get_init_pid”

What storage pool driver is in use?

Anything useful in /var/log/incus/incusd.log?

cat /var/log/incus/incusd.log
time=“2024-01-01T02:36:31-05:00” level=warning msg=“Dqlite: attempt 1: server 84.17.40.18:8443: no known leader”
time=“2024-01-01T02:36:31-05:00” level=warning msg=“Dqlite: attempt 1: server 84.17.40.18:8443: no known leader”
time=“2024-01-01T03:47:39-05:00” level=error msg=“Failed starting instance” action=start created=“2024-01-01 07:45:48.24563635 +0000 UTC” ephemeral=false instance=WPZ-DATABASE2024 instanceType=container project=default stateful=false used=“2024-01-01 07:45:51.043636476 +0000 UTC”
time=“2024-01-01T03:53:24-05:00” level=error msg=“Failed starting instance” action=start created=“2023-12-31 06:46:34.087557759 +0000 UTC” ephemeral=false instance=WP-WARHAPPENS2024 instanceType=container project=default stateful=false used=“2023-12-31 06:46:41.133819429 +0000 UTC”
time=“2024-01-01T03:53:42-05:00” level=error msg=“Failed starting instance” action=start created=“2023-12-31 06:46:34.087557759 +0000 UTC” ephemeral=false instance=WP-WARHAPPENS2024 instanceType=container project=default stateful=false used=“2024-01-01 08:53:24.443278498 +0000 UTC”
time=“2024-01-01T03:54:12-05:00” level=error msg=“Failed starting instance” action=start created=“2023-12-31 06:46:34.087557759 +0000 UTC” ephemeral=false instance=WP-WARHAPPENS2024 instanceType=container project=default stateful=false used=“2024-01-01 08:53:42.039945382 +0000 UTC”
time=“2024-01-01T04:03:05-05:00” level=error msg=“Failed starting instance” action=start created=“2023-12-31 06:46:34.087557759 +0000 UTC” ephemeral=false instance=WP-WARHAPPENS2024 instanceType=container project=default stateful=false used=“2024-01-01 08:54:12.063983709 +0000 UTC”
time=“2024-01-01T04:04:55-05:00” level=error msg=“Failed starting instance” action=start created=“2024-01-01 08:00:58.641996981 +0000 UTC” ephemeral=false instance=WP-SPACEWATCH2024 instanceType=container project=default stateful=false used=“2024-01-01 08:01:01.482134493 +0000 UTC”

cluster list
±-----±-------------------------±-----------------±-------------±---------------±------------±-------±------------------+
| NAME | URL | ROLES | ARCHITECTURE | FAILURE DOMAIN | DESCRIPTION | STATE | MESSAGE |
±-----±-------------------------±-----------------±-------------±---------------±------------±-------±------------------+
| Q1 | https://84.17.40.18:8443 | database-standby | x86_64 | default | | ONLINE | Fully operational |
±-----±-------------------------±-----------------±-------------±---------------±------------±-------±------------------+
| Q2 | https://84.17.40.19:8443 | database | x86_64 | default | | ONLINE | Fully operational |
±-----±-------------------------±-----------------±-------------±---------------±------------±-------±------------------+
| Q3 | https://84.17.40.20:8443 | database | x86_64 | default | | ONLINE | Fully operational |
±-----±-------------------------±-----------------±-------------±---------------±------------±-------±------------------+
| Q4 | https://84.17.40.21:8443 | database-leader | x86_64 | default | | ONLINE | Fully operational |
| | | database | | | | | |

Some more info. There are copied Containers from LXD, so may there is something with them.

cat console.log
systemd 237 running in system mode. (+PAM +AUDIT +SELINUX +IMA +APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD -IDN2 +IDN -PCRE2 default-hierarchy=hybrid)
Detected virtualization lxc.
Detected architecture x86-64.

Welcome to Ubuntu 18.04.6 LTS!

Set hostname to .
Initializing machine ID from random generator.
Failed to read AF_UNIX datagram queue length, ignoring: No such file or directory
File /lib/systemd/system/systemd-journald.service:36 configures an IP firewall (IPAddressDeny=any), but the local system does not support BPF/cgroup based firewalling.
Proceeding WITHOUT firewalling in effect! (This warning is only shown for the first loaded unit using IP firewalling.)
[ OK ] Started Dispatch Password Requests to Console Directory Watch.
[ OK ] Reached target Swap.
[ OK ] Started Forward Password Requests to Wall Directory Watch.
[ OK ] Reached target Local Encrypted Volumes.
[UNSUPP] Starting of Arbitrary Executable Fi…tem Automount Point not supported.
[ OK ] Reached target Remote File Systems.
system.slice: Failed to reset devices.list: Operation not permitted
[ OK ] Created slice System Slice.
[ OK ] Listening on udev Kernel Socket.
[ OK ] Listening on Journal Socket.
systemd-sysctl.service: Failed to reset devices.list: Operation not permitted
Starting Apply Kernel Variables…
keyboard-setup.service: Failed to reset devices.list: Operation not permitted
Starting Set the console keyboard layout…
[ OK ] Listening on Network Service Netlink Socket.
[ OK ] Listening on Syslog Socket.
user.slice: Failed to reset devices.list: Operation not permitted
[ OK ] Created slice User and Session Slice.
[ OK ] Reached target Slices.
[ OK ] Reached target Paths.
[ OK ] Listening on udev Control Socket.
systemd-udev-trigger.service: Failed to reset devices.list: Operation not permitted
Starting udev Coldplug all Devices…
[ OK ] Listening on Journal Socket (/dev/log).
systemd-journald.service: Failed to reset devices.list: Operation not permitted
Starting Journal Service…
systemd-tmpfiles-setup-dev.service: Failed to reset devices.list: Operation not permitted
Starting Create Static Device Nodes in /dev…
[ OK ] Listening on /dev/initctl Compatibility Named Pipe.
[ OK ] Started Apply Kernel Variables.
[ OK ] Started Create Static Device Nodes in /dev.
systemd-udevd.service: Failed to reset devices.list: Operation not permitted
Starting udev Kernel Device Manager…
[ OK ] Started Journal Service.
Starting Flush Journal to Persistent Storage…
[ OK ] Started udev Kernel Device Manager.
[ OK ] Started Set the console keyboard layout.
[ OK ] Reached target Local File Systems (Pre).
[ OK ] Reached target Local File Systems.
Starting Set console font and keymap…
Starting Network Service…
[ OK ] Started Set console font and keymap.
[ OK ] Started Flush Journal to Persistent Storage.
Starting Create Volatile Files and Directories…
[ OK ] Started Network Service.
[ OK ] Started Create Volatile Files and Directories.
[ OK ] Reached target System Time Synchronized.
Starting Update UTMP about System Boot/Shutdown…
Starting Network Name Resolution…
[ OK ] Started Update UTMP about System Boot/Shutdown.
[ OK ] Started Network Name Resolution.
[ OK ] Reached target Network.
[ OK ] Reached target Host and Network Name Lookups.
[ OK ] Started udev Coldplug all Devices.
[ OK ] Reached target System Initialization.
[ OK ] Started Daily apt download activities.
[ OK ] Started Message of the Day.
[ OK ] Listening on D-Bus System Message Bus Socket.
[ OK ] Reached target Sockets.
[ OK ] Reached target Basic System.
Starting Dispatcher daemon for systemd-networkd…
[ OK ] Started Regular background program processing daemon.
[ OK ] Started Daily apt upgrade and clean activities.
[ OK ] Started D-Bus System Message Bus.
Starting System Logging Service…
Starting Permit User Sessions…
Starting Login Service…
[ OK ] Started Daily Cleanup of Temporary Directories.
[ OK ] Reached target Timers.
[ OK ] Started System Logging Service.
[ OK ] Started Permit User Sessions.
[ OK ] Created slice system-getty.slice.
[ OK ] Started Console Getty.
[ OK ] Reached target Login Prompts.
Starting Hostname Service…
[ OK ] Started Login Service.
[ OK ] Started Hostname Service.
[ OK ] Started Dispatcher daemon for systemd-networkd.
[ OK ] Reached target Multi-User System.
[ OK ] Reached target Graphical Interface.
Starting Update UTMP about System Runlevel Changes…
[ OK ] Started Update UTMP about System Runlevel Changes.

Ubuntu 18.04.6 LTS WP-SPACEWATCH2024 console

WP-SPACEWATCH2024 login: [ OK ] Stopped target Host and Network Name Lookups.
[ OK ] Stopped target Timers.
[ OK ] Reached target Unmount All Filesystems.
[ OK ] Stopped Message of the Day.
[ OK ] Removed slice system-getty.slice.
[ OK ] Stopped Daily Cleanup of Temporary Directories.
[ OK ] Stopped Daily apt upgrade and clean activities.
[ OK ] Stopped Daily apt download activities.
[ OK ] Stopped target System Time Synchronized.
[ OK ] Stopped target Graphical Interface.
[ OK ] Stopped target Multi-User System.
[ OK ] Stopped target Login Prompts.
Stopping Console Getty…
Stopping System Logging Service…
Stopping D-Bus System Message Bus…
Stopping Dispatcher daemon for systemd-networkd…
Stopping Regular background program processing daemon…
Stopping Login Service…
[ OK ] Stopped Dispatcher daemon for systemd-networkd.
[ OK ] Stopped Regular background program processing daemon.
[ OK ] Stopped System Logging Service.
[ OK ] Stopped Login Service.
[ OK ] Stopped D-Bus System Message Bus.
[ OK ] Stopped Console Getty.
Stopping Permit User Sessions…
[ OK ] Stopped Permit User Sessions.
[ OK ] Stopped target Basic System.
[ OK ] Stopped target Sockets.
[ OK ] Closed D-Bus System Message Bus Socket.
[ OK ] Closed Syslog Socket.
[ OK ] Stopped target Paths.
[ OK ] Stopped target Slices.
[ OK ] Removed slice User and Session Slice.
[ OK ] Stopped target System Initialization.
[ OK ] Stopped target Local Encrypted Volumes.
[ OK ] Stopped Forward Password Requests to Wall Directory Watch.
[ OK ] Stopped Dispatch Password Requests to Console Directory Watch.
Stopping Update UTMP about System Boot/Shutdown…
[ OK ] Stopped target Swap.
[ OK ] Stopped target Network.
Stopping Network Name Resolution…
[ OK ] Stopped target Remote File Systems.
[ OK ] Stopped Network Name Resolution.
Stopping Network Service…
[ OK ] Stopped Update UTMP about System Boot/Shutdown.
[ OK ] Stopped Create Volatile Files and Directories.
[ OK ] Stopped target Local File Systems.
[ OK ] Stopped target Local File Systems (Pre).
[ OK ] Stopped Create Static Device Nodes in /dev.
[ OK ] Stopped Network Service.
[ OK ] Stopped Apply Kernel Variables.
[ OK ] Reached target Shutdown.
[ OK ] Reached target Final Step.
Starting Halt…

root@Q1:/var/log/incus/WP-SPACEWATCH2024# cat lxc.conf
lxc.log.file = /var/log/incus/WP-SPACEWATCH2024/lxc.log
lxc.log.level = warn
lxc.console.buffer.size = auto
lxc.console.size = auto
lxc.console.logfile = /var/log/incus/WP-SPACEWATCH2024/console.log
lxc.mount.auto = proc:rw sys:rw cgroup:mixed
lxc.autodev = 1
lxc.pty.max = 1024
lxc.mount.entry = /dev/fuse dev/fuse none bind,create=file,optional 0 0
lxc.mount.entry = /dev/net/tun dev/net/tun none bind,create=file,optional 0 0
lxc.mount.entry = /proc/sys/fs/binfmt_misc proc/sys/fs/binfmt_misc none rbind,create=dir,optional 0 0
lxc.mount.entry = /sys/fs/fuse/connections sys/fs/fuse/connections none rbind,create=dir,optional 0 0
lxc.mount.entry = /sys/fs/pstore sys/fs/pstore none rbind,create=dir,optional 0 0
lxc.mount.entry = /sys/kernel/config sys/kernel/config none rbind,create=dir,optional 0 0
lxc.mount.entry = /sys/kernel/debug sys/kernel/debug none rbind,create=dir,optional 0 0
lxc.mount.entry = /sys/kernel/security sys/kernel/security none rbind,create=dir,optional 0 0
lxc.mount.entry = /sys/kernel/tracing sys/kernel/tracing none rbind,create=dir,optional 0 0
lxc.mount.entry = /dev/mqueue dev/mqueue none rbind,create=dir,optional 0 0
lxc.include = /opt/incus/share/lxc/config//common.conf.d/
lxc.arch = linux64
lxc.hook.version = 1
lxc.hook.pre-start = /proc/2738/exe callhook /var/lib/incus “default” “WP-SPACEWATCH2024” start
lxc.hook.stop = /opt/incus/bin/incusd callhook /var/lib/incus “default” “WP-SPACEWATCH2024” stopns
lxc.hook.post-stop = /opt/incus/bin/incusd callhook /var/lib/incus “default” “WP-SPACEWATCH2024” stop
lxc.tty.max = 0
lxc.uts.name = WP-SPACEWATCH2024
lxc.mount.entry = /var/lib/incus/guestapi dev/incus none bind,create=dir 0 0
lxc.apparmor.profile = incus-WP-SPACEWATCH2024_</var/lib/incus>//&:incus-WP-SPACEWATCH2024_:
lxc.seccomp.profile = /var/lib/incus/security/seccomp/WP-SPACEWATCH2024
lxc.idmap = u 0 1000000 1000000000
lxc.idmap = g 0 1000000 1000000000
lxc.mount.auto = shmounts:/var/lib/incus/shmounts/WP-SPACEWATCH2024:/dev/.incus-mounts
lxc.net.0.type = phys
lxc.net.0.name = eth0
lxc.net.0.flags = up
lxc.net.0.link = vetha73f8f3e
lxc.net.0.hwaddr = 00:16:3e:61:27:49
lxc.rootfs.path = dir:/var/lib/incus/storage-pools/default/containers/WP-SPACEWATCH2024/rootfs
root@Q1:/var/log/incus/WP-SPACEWATCH2024# cat lxc.log
lxc WP-SPACEWATCH2024 20240101090455.639 WARN conf - …/src/lxc/conf.c:lxc_map_ids:3621 - newuidmap binary is missing
lxc WP-SPACEWATCH2024 20240101090455.639 WARN conf - …/src/lxc/conf.c:lxc_map_ids:3627 - newgidmap binary is missing
lxc WP-SPACEWATCH2024 20240101090455.640 WARN conf - …/src/lxc/conf.c:lxc_map_ids:3621 - newuidmap binary is missing
lxc WP-SPACEWATCH2024 20240101090455.640 WARN conf - …/src/lxc/conf.c:lxc_map_ids:3627 - newgidmap binary is missing
lxc WP-SPACEWATCH2024 20240101090455.641 WARN cgfsng - …/src/lxc/cgroups/cgfsng.c:fchowmodat:1619 - No such file or directory - Failed to fchownat(40, memory.oom.group, 1000000000, 0, AT_EMPTY_PATH | AT_SYMLINK_NOFOLLOW )
lxc WP-SPACEWATCH2024 20240101090455.770 ERROR conf - …/src/lxc/conf.c:run_buffer:322 - Script exited with status 32
lxc WP-SPACEWATCH2024 20240101090455.770 ERROR conf - …/src/lxc/conf.c:lxc_setup:4437 - Failed to run mount hooks
lxc WP-SPACEWATCH2024 20240101090455.771 ERROR start - …/src/lxc/start.c:do_start:1272 - Failed to setup container “WP-SPACEWATCH2024”
lxc WP-SPACEWATCH2024 20240101090455.771 ERROR sync - …/src/lxc/sync.c:sync_wait:34 - An error occurred in another process (expected sequence number 4)
lxc WP-SPACEWATCH2024 20240101090455.778 WARN network - …/src/lxc/network.c:lxc_delete_network_priv:3631 - Failed to rename interface with index 0 from “eth0” to its initial name “vetha73f8f3e”
lxc WP-SPACEWATCH2024 20240101090455.778 ERROR lxccontainer - …/src/lxc/lxccontainer.c:wait_on_daemonized_start:878 - Received container state “ABORTING” instead of “RUNNING”
lxc WP-SPACEWATCH2024 20240101090455.778 ERROR start - …/src/lxc/start.c:__lxc_start:2107 - Failed to spawn container “WP-SPACEWATCH2024”
lxc WP-SPACEWATCH2024 20240101090455.778 WARN start - …/src/lxc/start.c:lxc_abort:1036 - No such process - Failed to send SIGKILL via pidfd 41 for process 27343
lxc 20240101090455.930 ERROR af_unix - …/src/lxc/af_unix.c:lxc_abstract_unix_recv_fds_iov:218 - Connection reset by peer - Failed to receive response
lxc 20240101090455.930 ERROR commands - …/src/lxc/commands.c:lxc_cmd_rsp_recv_fds:128 - Failed to receive file descriptors for command “get_init_pid”

Ok, found the reason containers moved from LXD did not want to start there was LXD and LXCFS in /var/lib. I delete these and they can up perfect.

Ah, that’s interesting, it’s not something I ran into before and I’m a bit unsure how those would cause conflicts, but I’m glad you got those back online.