I have incus cluster of 3 nodes, debian 12 clean, ZFS zfs-2.1.11-1, Incus 6.0.2, each with ZFS pool, locally to each node. incusd on node1 not responding, can’t do any operations.
from node2 :
root@dcx02:~# incus cluster list -f compact
NAME URL ROLES ARCHITECTURE FAILURE DOMAIN DESCRIPTION STATUS MESSAGE
dcx01 https://192.168.248.51:8443 database x86_64 default ONLINE Fully operational
dcx02 https://192.168.248.52:8443 database x86_64 default ONLINE Fully operational
dcx03 https://192.168.248.53:8443 database-leader x86_64 default ONLINE Fully operational
database
Tried systemctl restart incus.service incus.socket on node1:
journalctl -xeu incus.service:
The unit incus.service completed and consumed the indicated resources.
Nov 24 09:58:07 dcx01 systemd[1]: incus.service: Found left-over process 3326271 (zfs) in control group while starting unit. Ignoring.
Nov 24 09:58:07 dcx01 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies.
Nov 24 09:58:07 dcx01 systemd[1]: Starting incus.service - Incus - Daemon…
Subject: A start job for unit incus.service has begun execution
Defined-By: systemd
Support: Debian -- User Support
A start job for unit incus.service has begun execution.
no logs in /var/log/incus/incusd.log
UPDATE :
trien to restart incusd again, then :
ps fauxww | grep -C 5 zfs :
root 3074728 0.0 0.0 7296 4224 pts/4 Ss 09:04 0:00 _ /bin/bash
root 3698822 900 0.0 15036 8040 pts/4 R+ 11:09 0:00 _ ps fauxww
root 3698823 0.0 0.0 6332 2256 pts/4 S+ 11:09 0:00 _ grep -C 5 zfs
root 3687006 2.5 0.0 7243168 185528 ? Ssl 11:07 0:03 incusd --group incus-admin --logfile /var/log/incus/incusd.log
root 3687141 0.0 0.0 14744 6312 ? D 11:07 0:00 _ zfs set mountpoint=legacy setuid=on exec=on devices=on acltype=posixacl xattr=sa relatime=on tank0-dcx1hd
root 3687008 0.0 0.0 6677192 43368 ? Ssl 11:07 0:00 incusd waitready --timeout=600
it seems some containers exceed storage quota , and seem not respcting quota in profile which should be 32GB
No host reboot yet, as the containers seem running and serving, and I’m afraid if incusd won’t start when host reboot.
should I reboot the host, and if incus not started somehow, what option I have to recover instances ? Thank you