Upgrade to 6.2, two servers lost Incus all together

Incus
1

So on my cluster with 8 servers, 2 lost their Incus altogether. Even the directories are gone. Like everything never existed after upgrade. They are running Ubuntu 22.04.4 LTS. Luckily I don’t have mission-critical apps in this cluster yet, but I have another that does. So presently this cluster is dead
incus version
Client version: 6.2
Server version: unreachable
From one of the Servers that has incus
INSERT INTO raft_nodes VALUES(1,‘192.168.1.18:8443’,1,‘RRSOC’);
INSERT INTO raft_nodes VALUES(2,‘192.168.1.15:8443’,2,‘rrterabk’);
INSERT INTO raft_nodes VALUES(3,‘192.168.1.50:8443’,0,‘RRAI’);
INSERT INTO raft_nodes VALUES(4,‘192.168.1.26:8443’,0,‘RRDB2’);
INSERT INTO raft_nodes VALUES(5,‘192.168.1.25:8443’,1,‘RRTEAMS’);
INSERT INTO raft_nodes VALUES(6,‘192.168.1.24:8443’,2,‘mail.rrmails.com’);
INSERT INTO raft_nodes VALUES(7,‘192.168.1.22:8443’,2,‘rrcloud’);
INSERT INTO raft_nodes VALUES(8,‘192.168.1.27:8443’,0,‘Q5’);
The missing ones are mail.rrmails.com and rrcloud.
Very strange everything is gone from them, even history.
Any ideas? Like I said it is not mission critical but it is a bit alarming that this happened so I figure you guys might want to investigate.

Let me know if you need anything?

2

Can you post your /var/log/apt/history.log for those servers?

3

cat /var/log/apt/history.log

Start-Date: 2024-07-01 06:08:33
Commandline: /usr/bin/unattended-upgrade
Upgrade: libcdio19:amd64 (2.1.0-3build1, 2.1.0-3ubuntu0.2)
End-Date: 2024-07-01 06:08:34

Start-Date: 2024-07-02 06:37:54
Commandline: /usr/bin/unattended-upgrade
Upgrade: libespeak-ng1:amd64 (1.50+dfsg-10, 1.50+dfsg-10ubuntu0.1), espeak-ng-data:amd64 (1.50+dfsg-10, 1.50+dfsg-10ubuntu0.1)End-Date: 2024-07-02 06:37:55

Start-Date: 2024-07-02 06:38:00
Commandline: /usr/bin/unattended-upgrade
Upgrade: openssh-client:amd64 (1:8.9p1-3ubuntu0.7, 1:8.9p1-3ubuntu0.10), openssh-server:amd64 (1:8.9p1-3ubuntu0.7, 1:8.9p1-3ubuntu0.10), openssh-sftp-server:amd64 (1:8.9p1-3ubuntu0.7, 1:8.9p1-3ubuntu0.10)
End-Date: 2024-07-02 06:38:05

Start-Date: 2024-07-04 06:19:16
Commandline: /usr/bin/unattended-upgrade
Upgrade: openvpn:amd64 (2.5.9-0ubuntu0.22.04.2, 2.5.9-0ubuntu0.22.04.3)
End-Date: 2024-07-04 06:19:19

Start-Date: 2024-07-08 09:35:27
Commandline: apt upgrade
Requested-By: rr2020 (1001)
Upgrade: bind9-host:amd64 (1:9.18.18-0ubuntu0.22.04.2, 1:9.18.24-0ubuntu0.22.04.1), libfprint-2-2:amd64 (1:1.94.3+tod1-0ubuntu2~22.04.06, 1:1.94.3+tod1-0ubuntu2~22.04.07), gjs:amd64 (1.72.2-0ubuntu2, 1.72.4-0ubuntu0.22.04.3~really.is.1.72.2.0ubuntu2),
libgjs0g:amd64 (1.72.2-0ubuntu2, 1.72.4-0ubuntu0.22.04.3~really.is.1.72.2.0ubuntu2), tzdata:amd64 (2024a-0ubuntu0.22.04, 2024a-0ubuntu0.22.04.1), xxd:amd64 (2:8.2.3995-1ubuntu2.16, 2:8.2.3995-1ubuntu2.17), libmutter-10-0:amd64 (42.9-0ubuntu7.1, 42.9-0ubuntu8), vim-common:amd64 (2:8.2.3995-1ubuntu2.16, 2:8.2.3995-1ubuntu2.17), gnome-remote-desktop:amd64 (42.9-0ubuntu0.22.04.1, 42.9-0ubuntu0.22.04.2), mutter-common:amd64 (42.9-0ubuntu7.1, 42.9-0ubuntu8), ubuntu-pro-client-l10n:amd64 (31.2.3~22.04, 32.3.1~22.04), gir1.2-mutter-10:amd64 (42.9-0ubuntu7.1, 42.9-0ubuntu8), vim-tiny:amd64 (2:8.2.3995-1ubuntu2.16, 2:8.2.3995-1ubuntu2.17), bind9-dnsutils:amd64 (1:9.18.18-0ubuntu0.22.04.2, 1:9.18.24-0ubuntu0.22.04.1), snapd:amd64 (2.62+22.04, 2.63+22.04), linux-firmware:amd64 (20220329.git681281e4-0ubuntu3.30, 20220329.git681281e4-0ubuntu3.31), timeshift:amd64 (21.09.1-1, 21.09.1-1ubuntu1), bind9-libs:amd64 (1:9.18.18-0ubuntu0.22.04.2, 1:9.18.24-0ubuntu0.22.04.1), ubuntu-advantage-tools:amd64 (31.2.3~22.04, 32.3.1~22.04), ubuntu-pro-client:amd64 (31.2.3~22.04, 32.3.1~22.04)
End-Date: 2024-07-08 09:37:14
root@rrcloud:/var/lib#

cat /var/log/apt/history.log
Start-Date: 2024-07-0bash: oot@rrcloud:/var/lib#: No such file or directory
1 06:08:33
Croot@rrmails:/var/lib#
root@rrmails:/var/lib# Start-Date: 2024-07-01 06:08:33
ommandline: /usr/bin/unattended-upgrade
Upgrade: libcdio19:amd64 (2.1.0-3build1, 2.1.0-3ubuntu0.2)
End-Date: 2024-07-01 06:08:34

Start-Date: 2024-07-02 06:37:54
Commandline: /usr/bin/unattended-upgrade
Upgrade: libespeak-ng1:amd64 (1.50+dfsg-10, 1.50+dfsg-10ubuntu0.1), espeak-ng-data:amd64 (1.50+dfsg-10,
1.50+dfsg-10ubuntu0.1)End-Date: 2024-07-02 06:37:55

Start-Date: 2024-07-02 06:38:00
Commandline: /usr/bin/unattended-upgrade
Upgrade: openssh-client:amd64 (1:8.9p1-3ubuntu0.7, 1:8.9p1-3ubuntu0.10), openssh-server:amd64 (1:8.9p1-3ubuntu0.7, 1:8.9p1-3ubuntu0.10), openssh-sftp-server:amd64 (1:8.9p1-3ubuntu0.7, 1:8.9p1-3ubuntu0.10)
End-Date: 2024-07-02 06:38:05

Start-Date: 2024-07-04 06:19:16
Commandline: /usr/bin/unattended-upgrade
Upgrade: openvpn:amd64 (2.5.9-0ubuntu0.22.04.2, 2.5.9-0ubuntu0.22.04.3)
End-Date: 2024-07-04 06:19:19

Start-Date: 2024-07-08 09:35:27
Commandline: apt upgrade
Requested-By: rr2020 (1001)
Upgrade: bind9-host:amd64 (1:9.18.18-0ubuntu0.22.04.2, 1:9.18.24-0ubuntu0.22.04.1), libfprint-2-2:amd64
(1:1.94.3+tod1-0ubuntu2~22.04.06, 1:1.94.3+tod1-0ubuntu2~22.04.07), gjs:amd64 (1.72.2-0ubuntu2, 1.72.40
ubuntu0.22.04.3~really.is.1.72.2.0ubuntu2),
libgjs0g:amd64 (1.72.2-0ubuntu2, 1.72.4-0ubuntu0.22.04.3~really.is.1.72.2.0ubuntu2), tzdata:amd64 (2024a-0ubuntu0.22.04, 2024a-0ubuntu0.22.04.1), xxd:amd64 (2:8.2.3995-1ubuntu2.16, 2:8.2.3995-1ubuntu2.17), libmutter-10-0:amd64 (42.9-0ubuntu7.1, 42.9-0ubuntu8), vim-common:amd64 (2:8.2.3995-1ubuntu2.16, 2:8.2.3995-1ubuntu2.17), gnome-remote-desktop:amd64 (42.9-0ubuntu0.22.04.1, 42.9-0ubuntu0.22.04.2), mutter-common:amd64 (42.9-0ubuntu7.1, 42.9-0ubuntu8), ubuntu-pro-client-l10n:amd64 (31.2.3~22.04, 32.3.1~22.04), gir1.2-mutter-10:amd64 (42.9-0ubuntStart-Date:: command not found
u7.1, 42.9-0ubuntu8), vim-tiny:amd64 (2:8.2.3995-1ubuntu2.16, 2:8.2.3995-1ubuntu2.17), bind9-dnsutils:amd64 (1:9.18.18-0ubuntu0root@rrmails:/var/lib# Commandline: /usr/bin/unattended-upgrade
.22.04.2, 1:9.18.24-0ubuntu0.22.04.1), snapd:amd64 (2.62+22.04, 2.63+22.04), linux-firmware:amd64 (20220329.git681281e4-0ubuntu3.30, 20220329.git681281e4-0ubuntu3.31), timeshift:amd64 (21.09.1-1, 21.09.1-1ubuntu1), bind9-libs:amd64 (1:9.18.18-0ubuntu0.22.04.2, 1:9.18.24-0ubuntu0.22.04.1), ubuntu-advantage-tools:amd64 (31.2.3~22.04, 32.3.1~22.04), ubuntu-pro-client:amd64 (31.2.3~22.04, 32.3.1~22.04)
End-Date: 2024-07-08 09:37:14
root@rrcloud:/var/lib#Commandline:: command not found
root@rrmails:/var/lib# Upgrade: libcdio19:amd64 (2.1.0-3build1, 2.1.0-3ubuntu0.2)
bash: syntax error near unexpected token (' root@rrmails:/var/lib# End-Date: 2024-07-01 06:08:34 End-Date:: command not found root@rrmails:/var/lib# root@rrmails:/var/lib# Start-Date: 2024-07-02 06:37:54 Start-Date:: command not found root@rrmails:/var/lib# Commandline: /usr/bin/unattended-upgrade Commandline:: command not found root@rrmails:/var/lib# Upgrade: libespeak-ng1:amd64 (1.50+dfsg-10, 1.50+dfsg-10ubuntu0.1), espeak-ng-data:amd64 (1.50+dfsg-10, 1.50+dfsg-10ubuntu0.1)End-Date: 2024-07-02 06:37:55 bash: syntax error near unexpected token (’
root@rrmails:/var/lib#
root@rrmails:/var/lib# Start-Date: 2024-07-02 06:38:00
Start-Date:: command not found
root@rrmails:/var/lib# Commandline: /usr/bin/unattended-upgrade
Commandline:: command not found
root@rrmails:/var/lib# Upgrade: openssh-client:amd64 (1:8.9p1-3ubuntu0.7, 1:8.9p1-3ubuntu0.10), openssh-server:amd64 (1:8.9p1-3ubuntu0.7, 1:8.9p1-3ubuntu0.10), openssh-sftp-server:amd64 (1:8.9p1-3ubuntu0.7, 1:8.9p1-3ubuntu0.10)
bash: syntax error near unexpected token `(’
root@rrmails:/var/lib# End-Date: 2024-07-02 06:38:05
End-Date:: command not found
root@rrmails:/var/lib#
root@rrmails:/var/lib# Start-Date: 2024-07-04 06:19:16
Start-Date:: command not found
root@rrmails:/var/lib# oot@rrcloud:/var/lib# cat /var/log/apt/history.log
Start-Dbash: oot@rrcloud:/var/lib#: No such file or directory
ate: 2root@rrmails:/var/lib#
root@rrmails:/var/lib# Start-Date: 2024-07-01 06:08:33
ommandline: /usr/bin/unattended-upgrade
Upgrade: libcdio19:amd64 (2.1.0-3build1, 2.1.0-3ubuntu0.2)
End-Date: 2024-07-01 06:08:34

Start-Date: 2024-07-02 06:37:54
Commandline: /usr/bin/unattended-upgrade
Upgrade: libespeak-ng1:amd64 (1.50+dfsg-10, 1.50+dfsg-10ubuntu0.1), espeak-ng-data:amd64 (1.50+dfsg-10,
1.50+dfsg-10ubuntu0.1)End-Date: 2024-07-02 06:37:55

Start-Date: 2024-07-02 06:38:00
Commandline: /usr/bin/unattended-upgrade
Upgrade: openssh-client:amd64 (1:8.9p1-3ubuntu0.7, 1:8.9p1-3ubuntu0.10), openssh-server:amd64 (1:8.9p1-3ubuntu0.7, 1:8.9p1-3ubuntu0.10), openssh-sftp-server:amd64 (1:8.9p1-3ubuntu0.7, 1:8.9p1-3ubuntu0.10)
End-Date: 2024-07-02 06:38:05

Start-Date: 2024-07-04 06:19:16
Commandline: /usr/bin/unattended-upgrade
Upgrade: openvpn:amd64 (2.5.9-0ubuntu0.22.04.2, 2.5.9-0ubuntu0.22.04.3)
End-Date: 2024-07-04 06:19:19

Start-Date: 2024-07-08 09:35:27
Commandline: apt upgrade
Requested-By: rr2020 (1001)
Upgrade: bind9-host:amd64 (1:9.18.18-0ubuntu0.22.04.2, 1:9.18.24-0ubuntu0.22.04.1), libfprint-2-2:amd64
(1:1.94.3+tod1-0ubuntu2~22.04.06, 1:1.94.3+tod1-0ubuntu2~22.04.07), gjs:amd64 (1.72.2-0ubuntu2, 1.72.4-0ubuntu0.22.04.3~really.is.1.72.2.0ubuntu2),
libgjs0g:amd64 (1.72.2-0ubuntu2, 1.72.4-0ubuntu0.22.04.3~really.is.1.72.2.0ubuntu2), tzdata:amd64 (2024a-0ubuntu0.22.04, 2024a-0ubuntu0.22.04.1), xxd:amd64 (2:8.2.3995-1ubuntu2.16, 2:8.2.3995-1ubuntu2.17), libmutter-10-0:amd64 (42.9-0ubuntu7.1, 42.9-0ubuntu8), vim-common:amd64 (2:8.2.3995-1ubuntu2.16, 2:8.2.3995-1ubuntu2.17), gnome-remote-desktop:amd64 (42.9-0ubuntu0.22.04.1, 42.9-0ubuntu0.22.04.2), mutter-common:amd64 (42.9-0ubuntu7.1, 42.9-0ubuntu8), ubuntu-pro-client-l10n:amd64 (31.2.3~22.04, 32.3.1~22.04), gir1.2-mutter-10:amd64 (42.9-0ubuntu7.1, 42.9-0ubuntu8), vim-tiny:amd64 (2:8.2.3995-1ubuntu2.16, 2:8.2.3995-1ubuntu2.17), bind9-dnsutils:amd64 (1:9.18.18-0ubuntu0.22.04.2, 1:9.18.24-0ubuntu0.22.04.1), snapd:amd64 (2.62+22.04, 2.63+22.04), linux-firmware:amd64 (20220329.git681281e4-0ubuntu3.30, 20220329.git681281e4-0ubuntu3.31), timeshift:amd64 (21.09.1-1, 21.09.1-1ubuntu1Start-Date:: command not found
), bind9-libs:amd64 (1:9.18.18-0ubuntu0.22.04.2, 1:9.18.24-0ubuntu0.22.04.1), ubuntu-advantage-tools:amd6root@rrmails:/var/lib# Commandline: /usr/bin/unattended-upgrade
4 (31.2.3~22.04, 32.3.1~22.04), ubuntu-pro-client:amd64 (31.2.3~22.04, 32.3.1~22.04)
End-Date: 2024-07-08 09:37:14
root@rrcloud:/var/lib#Commandline:: command not found
root@rrmails:/var/lib# Upgrade: libcdio19:amd64 (2.1.0-3build1, 2.1.0-3ubuntu0.2)
bash: syntax error near unexpected token (' root@rrmails:/var/lib# End-Date: 2024-07-01 06:08:34 End-Date:: command not found root@rrmails:/var/lib# root@rrmails:/var/lib# Start-Date: 2024-07-02 06:37:54 Start-Date:: command not found root@rrmails:/var/lib# Commandline: /usr/bin/unattended-upgrade Commandline:: command not found root@rrmails:/var/lib# Upgrade: libespeak-ng1:amd64 (1.50+dfsg-10, 1.50+dfsg-10ubuntu0.1), espeak-ng-data:amd64 (1.50+dfsg-10, 1.50+dfsg-10ubuntu0.1)End-Date: 2024-07-02 06:37:55 bash: syntax error near unexpected token (’
root@rrmails:/var/lib#
root@rrmails:/var/lib# Start-Date: 2024-07-02 06:38:00
Start-Date:: command not found
root@rrmails:/var/lib# Commandline: /usr/bin/unattended-upgrade
Commandline:: command not found
root@rrmails:/var/lib# Upgrade: openssh-client:amd64 (1:8.9p1-3ubuntu0.7, 1:8.9p1-3ubuntu0.10), openssh-server:amd64 (1:8.9p1-3ubuntu0.7, 1:8.9p1-3ubuntu0.10), openssh-sftp-server:amd64 (1:8.9p1-3ubuntu0.7, 1:8.9p1-3ubuntu0.10)
bash: syntax error near unexpected token (' root@rrmails:/var/lib# End-Date: 2024-07-02 06:38:05 End-Date:: command not found root@rrmails:/var/lib# root@rrmails:/var/lib# Start-Date: 2024-07-04 06:19:16 Start-Date:: command not found root@rrmails:/var/lib# Commandline: /usr/bin/unattended-upgrade Commandline:: command not found root@rrmails:/var/lib# Upgrade: openvpn:amd64 (2.5.9-0ubuntu0.22.04.2, 2.5.9-0ubuntu0.22.04.3) bash: syntax error near unexpected token (’
root@rrmails:/var/lib# End-Date: 2024-07-04 06:19:19
End-Date:: command not found
root@rrmails:/var/lib#
root@rrmails:/var/lib# Start-Date: 2024-07-08 09:35:27
Start-Date:: command not found
root@rrmails:/var/lib# Commandline: apt upgrade
Commandline:: command not found
root@rrmails:/var/lib# Requested-By: rr2020 (1001)
bash: syntax error near unexpected token (' root@rrmails:/var/lib# Upgrade: bind9-host:amd64 (1:9.18.18-0ubuntu0.22.04.2, 1:9.18.24-0ubuntu0.22.04.1), libfprint-2-2:amd64 (1:1.94.3+tod1-0ubuntu2~22.04.06, 1:1.94.3+tod1-0ubuntu2~22.04.07), gjs:amd64 (1.72.2-0ubuntu2, 1.72.4-0ubuntu0.22.04.3~really.is.1.72.2.0ubuntu2), bash: syntax error near unexpected token (’
root@rrmails:/var/lib# libgjs0g:amd64 (1.72.2-0ubuntu2, 1.72.4-0ubuntu0.22.04.3~really.is.1.72.2.0ubuntu2), tzdata:amd64 (2024a-0ubuntu0.22.04, 2024a-0ubuntu0.22.04.1), xxd:amd64 (2:8.2.3995-1ubuntu2.16, 2:8.2.3995-1ubuntu2.17), libmutter-10-0:amd64 (42.9-0ubuntu7.1, 42.9-0ubuntu8), vim-common:amd64 (2:8.2.3995-1ubuntu2.16, 2:8.2.3995-1ubuntu2.17), gnome-remote-desktop:amd64 (42.9-0ubuntu0.22.04.1, 42.9-0ubuntu0.22.04.2), mutter-common:amd64 (42.9-0ubuntu7.1, 42.9-0ubuntu8), ubuntu-pro-client-l10n:amd64 (31.2.3~22.04, 32.3.1~22.04), gir1.2-mutter-10:amd64 (42.9-0ubuntu7.1, 42.9-0ubuntu8), vim-tiny:amd64 (2:8.2.3995-1ubuntu2.16, 2:8.2.3995-1ubuntu2.17), bind9-dnsutils:amd64 (1:9.18.18-0ubuntu0.22.04.2, 1:9.18.24-0ubuntu0.22.04.1), snapd:amd64 (2.62+22.04, 2.63+22.04), linux-firmware:amd64 (20220329.git681281e4-0ubuntu3.30, 20220329.git681281e4-0ubuntu3.31), timeshift:amd64 (21.09.1-1, 21.09.1-1ubuntu1), bind9-libs:amd64 (1:9.18.18-0ubuntu0.22.04.2, 1:9.18.24-0ubuntu0.22.04.1), ubuntu-advantage-tools:amd64 (31.2.3~22.04, 32.3.1~22.04), ubuntu-pro-client:amd64 (31.2.3~22.04, 32.3.1~22.04)
bash: syntax error near unexpected token `1.72.2-0ubuntu2,’
root@rrmails:/var/lib# End-Date: 2024-07-08 09:37:14
End-Date:: command not found
root@rrmails:/var/lib#

4

Not seeing any mention of incus being removed in that log, so maybe it happened earlier in an older log?

5

Yes, it was removed sometime before. I am the only person that does updates/upgrades and have not done it in a while because I know I have to do all servers at once or have issues. I need to get those off the cluster on other servers so I can get it running because others are stuck and not able to upgrade either.
Worse case, I can remove/reinstall incus on all the machines but I would rather not do something when I don’t know the cause.

6

Can you show dpkg -l | grep incus on the broken servers?

7

nothing there on those two

on this one root@RRAI:/home/RR2020# dpkg -l | grep incus
iU incus 1:6.2-202406290147-ubuntu22.04
amd64 Incus - Container and virtualization daemon
iF incus-base 1:6.2-202406290147-ubuntu22.04
amd64 Incus - Container and virtualization daemon (container-only)
ii incus-client 1:6.2-202406290147-ubuntu22.04
amd64 Incus - Command line client

on another

root@RR5:/home/rr2020# dpkg -l | grep incus
iU incus 1:6.2-202406290147-ubuntu22.04 amd64 Incus - Container and
virtualization daemon
hF incus-base 1:6.2-202406290147-ubuntu22.04 amd64 Incus - Container and
virtualization daemon (container-only)
ii incus-client 1:6.2-202406290147-ubuntu22.04 amd64 Incus - Command line client
iU incus-ui-canonical 1:6.2-202406290147-ubuntu22.04 amd64 Incus - Web interface
(canonical)

8

BTW how can I turn RR5 server back to being standalone of cluster still running Incus. That way I can get those containers running. All the others in the other servers I can kill and start all over.

9

I was asking about the broken servers, so rrcloud and mail.rrmails.com in this case.

I indeed expect all the others to be stuck mid-upgrade in this scenario, that’s fine.

10

The broken ones show nothing literally.

root@rrcloud:/var/lib# dpkg -l | grep incus
root@rrcloud:/var/lib#

11

Okay, can you get the /var/log/apt/history.log on that one to figure out what happened to it?

12

There is No mention of Incus in either of these.

root@rrcloud:/var/lib# cat /var/log/apt/history.log

Start-Date: 2024-07-01 06:08:33
Commandline: /usr/bin/unattended-upgrade
Upgrade: libcdio19:amd64 (2.1.0-3build1, 2.1.0-3ubuntu0.2)
End-Date: 2024-07-01 06:08:34

Start-Date: 2024-07-02 06:37:54
Commandline: /usr/bin/unattended-upgrade
Upgrade: libespeak-ng1:amd64 (1.50+dfsg-10, 1.50+dfsg-10ubuntu0.1), espeak-ng-data:amd64 (1.50+dfsg-10, 1.50+dfsg-10ubuntu0.1)
End-Date: 2024-07-02 06:37:55

Start-Date: 2024-07-02 06:38:00
Commandline: /usr/bin/unattended-upgrade
Upgrade: openssh-client:amd64 (1:8.9p1-3ubuntu0.7, 1:8.9p1-3ubuntu0.10), openssh-server:amd64 (1:8.9p1-3ubuntu0.7, 1:8.9p1-3ubuntu0.10), openssh-sftp-server:amd64 (1:8.9p1-3ubuntu0.7, 1:8.9p1-3ubuntu0.10)
End-Date: 2024-07-02 06:38:05

Start-Date: 2024-07-04 06:19:16
Commandline: /usr/bin/unattended-upgrade
Upgrade: openvpn:amd64 (2.5.9-0ubuntu0.22.04.2, 2.5.9-0ubuntu0.22.04.3)
End-Date: 2024-07-04 06:19:19

Start-Date: 2024-07-08 09:35:27
Commandline: apt upgrade
Requested-By: rr2020 (1001)
Upgrade: bind9-host:amd64 (1:9.18.18-0ubuntu0.22.04.2, 1:9.18.24-0ubuntu0.22.04.1), libfprint-2-2:amd64 (1:1.94.3+tod1-0ubuntu2~22.04.06, 1:1.94.3+tod1-0ubuntu2~22.04.07), gjs:amd64 (1.72.2-0ubuntu2, 1.72.4-0ubuntu0.22.04.3~really.is.1.72.2.0ubuntu2), libgjs0g:amd64 (1.72.2-0ubuntu2, 1.72.4-0ubuntu0.22.04.3~really.is.1.72.2.0ubuntu2), tzdata:amd64 (2024a-0ubuntu0.22.04, 2024a-0ubuntu0.22.04.1), xxd:amd64 (2:8.2.3995-1ubuntu2.16, 2:8.2.3995-1ubuntu2.17), libmutter-10-0:amd64 (42.9-0ubuntu7.1, 42.9-0ubuntu8), vim-common:amd64 (2:8.2.3995-1ubuntu2.16, 2:8.2.3995-1ubuntu2.17), gnome-remote-desktop:amd64 (42.9-0ubuntu0.22.04.1, 42.9-0ubuntu0.22.04.2), mutter-common:amd64 (42.9-0ubuntu7.1, 42.9-0ubuntu8), ubuntu-pro-client-l10n:amd64 (31.2.3~22.04, 32.3.1~22.04), gir1.2-mutter-10:amd64 (42.9-0ubuntu7.1, 42.9-0ubuntu8), vim-tiny:amd64 (2:8.2.3995-1ubuntu2.16, 2:8.2.3995-1ubuntu2.17), bind9-dnsutils:amd64 (1:9.18.18-0ubuntu0.22.04.2, 1:9.18.24-0ubuntu0.22.04.1), snapd:amd64 (2.62+22.04, 2.63+22.04), linux-firmware:amd64 (20220329.git681281e4-0ubuntu3.30, 20220329.git681281e4-0ubuntu3.31), timeshift:amd64 (21.09.1-1, 21.09.1-1ubuntu1), bind9-libs:amd64 (1:9.18.18-0ubuntu0.22.04.2, 1:9.18.24-0ubuntu0.22.04.1), ubuntu-advantage-tools:amd64 (31.2.3~22.04, 32.3.1~22.04), ubuntu-pro-client:amd64 (31.2.3~22.04, 32.3.1~22.04)
End-Date: 2024-07-08 09:37:14

cat /var/log/apt/history.log

Start-Date: 2024-07-02 06:40:43
Commandline: /usr/bin/unattended-upgrade
Upgrade: libcdio19:amd64 (2.1.0-3build1, 2.1.0-3ubuntu0.2)
End-Date: 2024-07-02 06:40:44

Start-Date: 2024-07-02 06:40:49
Commandline: /usr/bin/unattended-upgrade
Upgrade: openssh-client:amd64 (1:8.9p1-3ubuntu0.7, 1:8.9p1-3ubuntu0.10), openssh-server:amd64 (1:8.9p1-3ubuntu0.7, 1:8.9p1-3ubuntu0.10), openssh-sftp-server:amd64 (1:8.9p1-3ubuntu0.7, 1:8.9p1-3ubuntu0.10)
End-Date: 2024-07-02 06:40:52

Start-Date: 2024-07-02 06:40:57
Commandline: /usr/bin/unattended-upgrade
Upgrade: libespeak-ng1:amd64 (1.50+dfsg-10, 1.50+dfsg-10ubuntu0.1), espeak-ng-data:amd64 (1.50+dfsg-10, 1.50+dfsg-10ubuntu0.1)
End-Date: 2024-07-02 06:40:58

Start-Date: 2024-07-03 06:26:37
Commandline: /usr/bin/unattended-upgrade
Upgrade: openvpn:amd64 (2.5.9-0ubuntu0.22.04.2, 2.5.9-0ubuntu0.22.04.3)
End-Date: 2024-07-03 06:26:40

Start-Date: 2024-07-08 09:36:56
Commandline: apt upgrade
Requested-By: rr2020 (1000)
Upgrade: bind9-host:amd64 (1:9.18.18-0ubuntu0.22.04.2, 1:9.18.24-0ubuntu0.22.04.1), libfprint-2-2:amd64 (1:1.94.3+tod1-0ubuntu2~22.04.06, 1:1.94.3+tod1-0ubuntu2~22.04.07), gjs:amd64 (1.72.2-0ubuntu2, 1.72.4-0ubuntu0.22.04.3~really.is.1.72.2.0ubuntu2), libgjs0g:amd64 (1.72.2-0ubuntu2, 1.72.4-0ubuntu0.22.04.3~really.is.1.72.2.0ubuntu2), tzdata:amd64 (2024a-0ubuntu0.22.04, 2024a-0ubuntu0.22.04.1), xserver-xorg-core:amd64 (2:21.1.4-2ubuntu1.7~22.04.10, 2:21.1.4-2ubuntu1.7~22.04.11), xxd:amd64 (2:8.2.3995-1ubuntu2.16, 2:8.2.3995-1ubuntu2.17), libmutter-10-0:amd64 (42.9-0ubuntu7.1, 42.9-0ubuntu8), vim-common:amd64 (2:8.2.3995-1ubuntu2.16, 2:8.2.3995-1ubuntu2.17), gnome-remote-desktop:amd64 (42.9-0ubuntu0.22.04.1, 42.9-0ubuntu0.22.04.2), mutter-common:amd64 (42.9-0ubuntu7.1, 42.9-0ubuntu8), xserver-xorg-legacy:amd64 (2:21.1.4-2ubuntu1.7~22.04.10, 2:21.1.4-2ubuntu1.7~22.04.11), ubuntu-pro-client-l10n:amd64 (31.2.3~22.04, 32.3.1~22.04), xserver-common:amd64 (2:21.1.4-2ubuntu1.7~22.04.10, 2:21.1.4-2ubuntu1.7~22.04.11), gir1.2-mutter-10:amd64 (42.9-0ubuntu7.1, 42.9-0ubuntu8), vim-tiny:amd64 (2:8.2.3995-1ubuntu2.16, 2:8.2.3995-1ubuntu2.17), bind9-dnsutils:amd64 (1:9.18.18-0ubuntu0.22.04.2, 1:9.18.24-0ubuntu0.22.04.1), snapd:amd64 (2.62+22.04, 2.63+22.04), linux-firmware:amd64 (20220329.git681281e4-0ubuntu3.30, 20220329.git681281e4-0ubuntu3.31), xserver-xephyr:amd64 (2:21.1.4-2ubuntu1.7~22.04.10, 2:21.1.4-2ubuntu1.7~22.04.11), bind9-libs:amd64 (1:9.18.18-0ubuntu0.22.04.2, 1:9.18.24-0ubuntu0.22.04.1), ubuntu-advantage-tools:amd64 (31.2.3~22.04, 32.3.1~22.04), ubuntu-pro-client:amd64 (31.2.3~22.04, 32.3.1~22.04)
End-Date: 2024-07-08 09:38:44

13

Okay, it’d be good to look at all the files in /var/log/apt/ and /var/log/dpkg/ to track down exactly what caused those packages to go away because it’s not something that incus itself could have done.

Does incus cluster list work? Normally that remains functional even on clusters that are waiting for servers to come online.

Can you also show incus admin sql global "SELECT * FROM nodes" as that one should definitely work even on a partially upgraded cluster.

14

incus cluster list hangs forever on all servers.

root@RRAI:/home/RR2020# incus admin sql global “SELECT * FROM nodes”

±—±-----------------±------------±------------------±-------±---------------±------------------------------------±------±-----±------------------+
| id | name | description | address | schema | api_extensions | heartbeat | state | arch | failure_domain_id |
±—±-----------------±------------±------------------±-------±---------------±------------------------------------±------±-----±------------------+
| 1 | RRSOC | | 192.168.1.18:8443 | 73 | 403 | 2024-07-08T12:42:14.769640576-04:00 | 0 | 2 | |
| 2 | rrterabk | | 192.168.1.15:8443 | 73 | 395 | 2024-05-18T15:17:36.538545837-04:00 | 0 | 2 | |
| 3 | RRAI | | 192.168.1.50:8443 | 73 | 403 | 2024-07-08T12:42:10.690750329-04:00 | 0 | 2 | |
| 4 | RRDB2 | | 192.168.1.26:8443 | 73 | 403 | 2024-07-08T12:42:08.157427231-04:00 | 0 | 2 | |
| 5 | RRTEAMS | | 192.168.1.25:8443 | 73 | 403 | 2024-07-08T12:42:11.405382772-04:00 | 0 | 2 | |
| 6 | mail.rrmails.com | | 192.168.1.24:8443 | 73 | 395 | 2024-05-19T07:27:43.795829475-04:00 | 0 | 2 | |
| 7 | rrcloud | | 192.168.1.22:8443 | 73 | 395 | 2024-05-18T15:50:17.006253106-04:00 | 0 | 2 | |
| 8 | Q5 | | 192.168.1.27:8443 | 73 | 403 | 2024-07-08T12:42:08.304167398-04:00 | 0 | 2 | |
±—±-----------------±------------±------------------±-------±---------------±------------------------------------±------±-----±------------------+
root@RRAI:/home/RR2020#

15

Those two got completely wipeout including all history. Except for cluster database on other machines.

16

This is what happens on other servers…

root@RRAI:/home/RR2020# apt update
Hit:1 Index of /ubuntu jammy InRelease
Hit:2 Index of /repos/stable/ubuntu/ jammy/ InRelease
Get:3 Index of /ubuntu jammy-updates InRelease [128 kB]
Get:4 Index of /repos/repoconfig/ubuntu/ jammy/ InRelease [1,285 B]
Get:5 Index of /ubuntu jammy-security InRelease [129 kB]
Hit:6 Index of /ubuntu jammy-backports InRelease
Get:7 Index of /repos/repoconfig/ubuntu/ jammy/ Packages [1,484 B]
Get:8 Index of /ubuntu jammy-updates/main i386 Packages [654 kB]
Get:9 Index of /ubuntu jammy-updates/main amd64 Packages [1,791 kB]
Get:10 Index of /ubuntu jammy-security/main i386 Packages [497 kB]
Get:11 Index of /ubuntu jammy-security/main amd64 Packages [1,583 kB]
Get:12 Index of /ubuntu jammy-updates/universe amd64 Packages [1,101 kB]
Get:13 Index of /ubuntu jammy-updates/universe i386 Packages [716 kB]
Get:14 Index of /ubuntu jammy-security/universe i386 Packages [616 kB]
Get:15 Index of /ubuntu jammy-updates/universe Translation-en [255 kB]
Get:16 Index of /ubuntu jammy-security/universe amd64 Packages [883 kB]
Hit:17 Index of /incus/stable/ jammy InRelease
Fetched 8,356 kB in 4s (2,187 kB/s)
Reading package lists… Done
Building dependency tree… Done
Reading state information… Done
11 packages can be upgraded. Run ‘apt list --upgradable’ to see them.
root@RRAI:/home/RR2020# apt list --upgradable
Listing… Done
python3-update-manager/jammy-updates,jammy-updates 1:22.04.20 all [upgradable from: 1:22.04.19]
ubuntu-desktop/jammy-updates 1.481.2 amd64 [upgradable from: 1.481.1]
ubuntu-minimal/jammy-updates 1.481.2 amd64 [upgradable from: 1.481.1]
ubuntu-standard/jammy-updates 1.481.2 amd64 [upgradable from: 1.481.1]
update-manager-core/jammy-updates,jammy-updates 1:22.04.20 all [upgradable from: 1:22.04.19]
update-manager/jammy-updates,jammy-updates 1:22.04.20 all [upgradable from: 1:22.04.19]
xserver-common/jammy-updates,jammy-updates 2:21.1.4-2ubuntu1.7~22.04.11 all [upgradable from: 2:21.1.4-2ubuntu1.7~22.04.10]
xserver-xephyr/jammy-updates 2:21.1.4-2ubuntu1.7~22.04.11 amd64 [upgradable from: 2:21.1.4-2ubuntu1.7~22.04.10]
xserver-xorg-core/jammy-updates 2:21.1.4-2ubuntu1.7~22.04.11 amd64 [upgradable from: 2:21.1.4-2ubuntu1.7~22.04.10]
xserver-xorg-legacy/jammy-updates 2:21.1.4-2ubuntu1.7~22.04.11 amd64 [upgradable from: 2:21.1.4-2ubuntu1.7~22.04.10]
root@RRAI:/home/RR2020# apt upgrade
Reading package lists… Done
Building dependency tree… Done
Reading state information… Done
Calculating upgrade… Done
The following package was automatically installed and is no longer required:
debian-archive-keyring
Use ‘sudo apt autoremove’ to remove it.

OpenSSH CVE-2024-6387 has been fixed for 22.04 LTS, 23.10 and 24.04 LTS.

RegreSSHion: Possible RCE Due To A Race Condition In Signal Handling.

For more details see: https://ubuntu.com/security/notices/USN-6859-1.

The following packages have been kept back:
python3-update-manager ubuntu-desktop ubuntu-desktop-minimal ubuntu-minimal ubuntu-standard update-manager update-manager-core
xserver-common xserver-xephyr xserver-xorg-core xserver-xorg-legacy
0 upgraded, 0 newly installed, 0 to remove and 11 not upgraded.
2 not fully installed or removed.
After this operation, 0 B of additional disk space will be used.
Do you want to continue? [Y/n] y
Setting up incus-base (1:6.2-202406290147-ubuntu22.04) …

Progress: [ 20%] [######################…]

17

incus admin sql global “UPDATE nodes SET api_extensions=403”
That should unblock things by pretending that the two dead servers were upgraded.

18

Yep, that did the trick… Thanks

Very mysterious how it disappeared from two servers, Ubuntu updates/upgrades must have killed it and then autoremove them.

19

Yeah, that’s where it’d be good to see the log entries of that happening to understand what package/upgrade may have caused this. Unless unattended-upgrade or something like that was very seriously misconfigured on the system, this kind of package deletion should not have been possible without apt or apt-get clearly showing you those packages being removed during an upgrade.

In any case, now you should use incus cluster remove --force on those two servers to have them out of the cluster, or you’ll be back to the same issue when upgrading to 6.3.

1 Like
20

Note that anything other than “ii” shows a broken / not-correctly-installed package. You can get the list of statuses from “dpkg-query -l | head -2”

Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend

Hence:

iU = you asked for it to be installed, and it has been Unpacked, but not yet configured
iF = you asked for it to be installed, but it’s only halF-configured
hF = you applied a hold on this package (which should prevent it being upgraded), and it’s also halF-configured

Without the logs from when it was last updated or attempted to be updated, it’s hard to know what went wrong with it.

Check with:

cat /var/log/apt/term.log
zcat /var/log/apt/term.log.1.gz  # repeat for .2.gz, .3.gz etc

Similarly, for older history you can do

zcat /var/log/apt/history.log.1.gz  # repeat for .2.gz, .3.gz etc