Incus 6.5 has been released

News
,
11

Thank you Stéphane again for the details.

I did the experiment twice with 3 node cluster (3x Ubuntu VMs under Hyper-V). In first look this approach works. However, there are some questions (at the bottom).

Setup update on each node

  • Put update of incus, incus-base, incus-client', incus-ui-canonical’ packages on hold using sudo apt-mark hold incus incus-base incus-client incus-ui-canonical
  • The shell script file was created and put in the /var/lib/incus-update/ folder:
#!/bin/sh
logger 'Incus autoupdate has been called'; \
  apt-get update && \
  logger 'Installing packages' && \
  apt-get install --allow-change-held-packages -y incus incus-base incus-client incus-ui-canonical && \
  logger -i -p info -t incus.update "incus packages have been updated" || \
  logger -i -p info -t incus.update "incus packages update has failed with code $?"
systemctl restart incus-lxcfs.service
  • The line INCUS_CLUSTER_UPDATE="/var/lib/incus-update/<script_name>" was added to the /etc/default/incus

Testing

  • Run script above on one of the node (neverminded if it database-leader or not).
  • Watch syslog on every other nodes via tail -f /var/log/syslog | grep incus

Results

  • Updating incus on one of the nodes triggers updating incus on other nodes by executing the script above.
  • The incus daemon restarts once on node where update process was initiated from console.
  • The incus daemon restarts two times on nodes where update process triggers by incus daemon.

Update logs
Node where updated initiated from the console

Sep  9 04:48:57 vm-01 systemd[1]: incus.service: Deactivated successfully.
Sep  9 04:48:57 vm-01 systemd[1]: incus.service: Unit process 2405 (dnsmasq) remains running after unit stopped.
Sep  9 04:48:57 vm-01 systemd[1]: incus.service: Consumed 3.180s CPU time.
Sep  9 04:48:57 vm-01 systemd[1]: incus.service: Found left-over process 2405 (dnsmasq) in control group while starting unit. Ignoring.
Sep  9 04:48:57 vm-01 incusd[4764]: time="2024-09-09T04:48:57Z" level=warning msg="Wait for other cluster nodes to upgrade their versions, cluster not started yet"
Sep  9 04:49:33 vm-01 kernel: [  409.293482] audit: type=1400 audit(1725857373.350:69): apparmor="STATUS" operation="profile_replace" profile="unconfined" name="incus_dnsmasq-lxdbr0_</var/lib/incus>" pid=4888 comm="apparmor_parser"
Sep  9 04:49:33 vm-01 dnsmasq[4889]: using only locally-known addresses for incus
Sep  9 04:49:33 vm-01 dnsmasq[4889]: using only locally-known addresses for incus
Sep  9 04:49:33 vm-01 dnsmasq-dhcp[4889]: read /var/lib/incus/networks/lxdbr0/dnsmasq.hosts/infra_dns-01.enp5s0
Sep  9 04:49:33 vm-01 dnsmasq-dhcp[4889]: read /var/lib/incus/networks/lxdbr0/dnsmasq.hosts/infra_dns-04.enp5s0
Sep  9 04:49:33 vm-01 dnsmasq-dhcp[4889]: read /var/lib/incus/networks/lxdbr0/dnsmasq.hosts/infra_dns-setup.enp5s0
Sep  9 04:49:33 vm-01 dnsmasq-dhcp[4889]: read /var/lib/incus/networks/lxdbr0/dnsmasq.hosts/pki_test.enp5s0
Sep  9 04:49:34 vm-01 incusd[4764]: time="2024-09-09T04:49:34Z" level=error msg="Failed to register device" device=bootstrap err="Storage volume \"dns_bootstrap\" in project \"infra\" of type \"custom\" does not exist on pool \"remote\": Storage volume not found" instance=dns-01 instanceType=container project=infra
Sep  9 04:49:34 vm-01 incusd[4764]: time="2024-09-09T04:49:34Z" level=error msg="Failed to register device" device=bootstrap err="Storage volume \"dns_bootstrap\" in project \"infra\" of type \"custom\" does not exist on pool \"remote\": Storage volume not found" instance=dns-04 instanceType=container project=infra
Sep  9 04:49:34 vm-01 incusd[4764]: time="2024-09-09T04:49:34Z" level=warning msg="No active cluster event listener clients"
Sep  9 04:49:34 vm-01 incusd[4764]: time="2024-09-09T04:49:34Z" level=error msg="Failed updating instance types" err="context canceled"
Sep  9 04:49:34 vm-01 incusd[4764]: time="2024-09-09T04:49:34Z" level=warning msg="Failed to update instance types: Get \"https://images.linuxcontainers.org/meta/instance-types/aws.yaml\": context canceled"
Sep  9 04:49:34 vm-01 incusd[4764]: time="2024-09-09T04:49:34Z" level=warning msg="Aborting heartbeat round" err="context canceled" mode=normal
Sep  9 04:49:34 vm-01 incusd[4764]: time="2024-09-09T04:49:34Z" level=warning msg="Rejecting heartbeat request as shutting down"
Sep  9 04:49:34 vm-01 incusd[4764]: time="2024-09-09T04:49:34Z" level=error msg="Failed cleaning up operations"
Sep  9 04:49:34 vm-01 incusd[4764]: time="2024-09-09T04:49:34Z" level=warning msg="Failed setting up event connection" err="Failed to begin transaction: sql: database is closed" remote="<VM2_IP>:46134"
Sep  9 04:49:34 vm-01 incusd[4764]: time="2024-09-09T04:49:34Z" level=warning msg="Rejecting heartbeat request as shutting down"
Sep  9 04:49:34 vm-01 incusd[4764]: time="2024-09-09T04:49:34Z" level=warning msg="Rejecting heartbeat request as shutting down"
Sep  9 04:49:34 vm-01 systemd[1]: incus.service: Deactivated successfully.
Sep  9 04:49:34 vm-01 systemd[1]: incus.service: Unit process 4889 (dnsmasq) remains running after unit stopped.
Sep  9 04:49:34 vm-01 systemd[1]: incus.service: Consumed 1.897s CPU time.
Sep  9 04:49:34 vm-01 systemd[1]: incus.service: Found left-over process 4889 (dnsmasq) in control group while starting unit. Ignoring.
Sep  9 04:49:40 vm-01 kernel: [  416.727408] audit: type=1400 audit(1725857380.782:70): apparmor="STATUS" operation="profile_replace" info="same as current profile, skipping" profile="unconfined" name="incus_dnsmasq-lxdbr0_</var/lib/incus>" pid=5038 comm="apparmor_parser"
Sep  9 04:49:40 vm-01 dnsmasq[5040]: using only locally-known addresses for incus
Sep  9 04:49:40 vm-01 dnsmasq[5040]: using only locally-known addresses for incus
Sep  9 04:49:40 vm-01 dnsmasq-dhcp[5040]: read /var/lib/incus/networks/lxdbr0/dnsmasq.hosts/infra_dns-01.enp5s0
Sep  9 04:49:40 vm-01 dnsmasq-dhcp[5040]: read /var/lib/incus/networks/lxdbr0/dnsmasq.hosts/infra_dns-04.enp5s0
Sep  9 04:49:40 vm-01 dnsmasq-dhcp[5040]: read /var/lib/incus/networks/lxdbr0/dnsmasq.hosts/infra_dns-setup.enp5s0
Sep  9 04:49:40 vm-01 dnsmasq-dhcp[5040]: read /var/lib/incus/networks/lxdbr0/dnsmasq.hosts/pki_test.enp5s0
Sep  9 04:49:41 vm-01 incusd[4938]: time="2024-09-09T04:49:41Z" level=error msg="Failed to register device" device=bootstrap err="Storage volume \"dns_bootstrap\" in project \"infra\" of type \"custom\" does not exist on pool \"remote\": Storage volume not found" instance=dns-01 instanceType=container project=infra
Sep  9 04:49:41 vm-01 incusd[4938]: time="2024-09-09T04:49:41Z" level=error msg="Failed to register device" device=bootstrap err="Storage volume \"dns_bootstrap\" in project \"infra\" of type \"custom\" does not exist on pool \"remote\": Storage volume not found" instance=dns-04 instanceType=container project=infra
Sep  9 04:49:43 vm-01 incusd[4938]: time="2024-09-09T04:49:43Z" level=warning msg="Cluster member isn't responding" name=vm-02
Sep  9 04:49:44 vm-01 incusd[4938]: time="2024-09-09T04:49:44Z" level=warning msg="Cluster member isn't responding" name=vm-02
Sep  9 04:50:06 vm-01 incusd[4938]: time="2024-09-09T04:50:06Z" level=warning msg="Rejecting heartbeat request as shutting down"
Sep  9 04:50:06 vm-01 incusd[4938]: time="2024-09-09T04:50:06Z" level=error msg="Failed cleaning up operations"
Sep  9 04:50:06 vm-01 incusd[4938]: time="2024-09-09T04:50:06Z" level=warning msg="Rejecting heartbeat request as shutting down"
Sep  9 04:50:06 vm-01 incusd[4938]: message repeated 2 times: [ time="2024-09-09T04:50:06Z" level=warning msg="Rejecting heartbeat request as shutting down"]
Sep  9 04:50:06 vm-01 systemd[1]: incus.service: Deactivated successfully.
Sep  9 04:50:06 vm-01 systemd[1]: incus.service: Unit process 5040 (dnsmasq) remains running after unit stopped.
Sep  9 04:50:06 vm-01 systemd[1]: incus.service: Consumed 1.623s CPU time.
Sep  9 04:50:06 vm-01 systemd[1]: var-lib-incus\x2dlxcfs.mount: Deactivated successfully.
Sep  9 04:50:06 vm-01 systemd[1]: incus-lxcfs.service: Main process exited, code=exited, status=1/FAILURE
Sep  9 04:50:06 vm-01 fusermount[5107]: /bin/fusermount: failed to unmount /var/lib/incus-lxcfs: Invalid argument
Sep  9 04:50:06 vm-01 systemd[1]: incus-lxcfs.service: Failed with result 'exit-code'.
Sep  9 04:50:06 vm-01 lxcfs[5109]: Starting LXCFS at /opt/incus/bin/lxcfs
Sep  9 04:50:06 vm-01 systemd[1]: incus.service: Found left-over process 5040 (dnsmasq) in control group while starting unit. Ignoring.
Sep  9 04:50:06 vm-01 kernel: [  442.751163] audit: type=1400 audit(1725857406.806:71): apparmor="STATUS" operation="profile_replace" info="same as current profile, skipping" profile="unconfined" name="incus_dnsmasq-lxdbr0_</var/lib/incus>" pid=5231 comm="apparmor_parser"
Sep  9 04:50:06 vm-01 dnsmasq[5233]: using only locally-known addresses for incus
Sep  9 04:50:06 vm-01 dnsmasq[5233]: using only locally-known addresses for incus
Sep  9 04:50:06 vm-01 dnsmasq-dhcp[5233]: read /var/lib/incus/networks/lxdbr0/dnsmasq.hosts/infra_dns-01.enp5s0
Sep  9 04:50:06 vm-01 dnsmasq-dhcp[5233]: read /var/lib/incus/networks/lxdbr0/dnsmasq.hosts/infra_dns-04.enp5s0
Sep  9 04:50:06 vm-01 dnsmasq-dhcp[5233]: read /var/lib/incus/networks/lxdbr0/dnsmasq.hosts/infra_dns-setup.enp5s0
Sep  9 04:50:06 vm-01 dnsmasq-dhcp[5233]: read /var/lib/incus/networks/lxdbr0/dnsmasq.hosts/pki_test.enp5s0
Sep  9 04:50:07 vm-01 incusd[5113]: time="2024-09-09T04:50:07Z" level=error msg="Failed to register device" device=bootstrap err="Storage volume \"dns_bootstrap\" in project \"infra\" of type \"custom\" does not exist on pool \"remote\": Storage volume not found" instance=dns-01 instanceType=container project=infra
Sep  9 04:50:07 vm-01 incusd[5113]: time="2024-09-09T04:50:07Z" level=error msg="Failed to register device" device=bootstrap err="Storage volume \"dns_bootstrap\" in project \"infra\" of type \"custom\" does not exist on pool \"remote\": Storage volume not found" instance=dns-04 instanceType=container project=infra
Sep  9 04:50:07 vm-01 incus.update[5269]: incus packages have been updated
Sep  9 04:50:07 vm-01 systemd[1]: var-lib-incus\x2dlxcfs.mount: Deactivated successfully.
Sep  9 04:50:07 vm-01 systemd[1]: incus-lxcfs.service: Main process exited, code=exited, status=1/FAILURE
Sep  9 04:50:07 vm-01 fusermount[5272]: /bin/fusermount: failed to unmount /var/lib/incus-lxcfs: Invalid argument
Sep  9 04:50:07 vm-01 systemd[1]: incus-lxcfs.service: Failed with result 'exit-code'.
Sep  9 04:50:07 vm-01 lxcfs[5274]: Starting LXCFS at /opt/incus/bin/lxcfs

Nodes where update triggered by the incus daemon trough INCUS_CLUSTER_UPDATE script

Sep  9 04:49:00 vm-02 incusd[1357]: time="2024-09-09T04:49:00Z" level=warning msg="Member is out-of-date with respect to other cluster members"
Sep  9 04:49:29 vm-02 systemd[1]: incus.service: Deactivated successfully.
Sep  9 04:49:29 vm-02 systemd[1]: incus.service: Unit process 2384 (dnsmasq) remains running after unit stopped.
Sep  9 04:49:29 vm-02 systemd[1]: incus.service: Unit process 4071 (incus.updater) remains running after unit stopped.
Sep  9 04:49:29 vm-02 systemd[1]: incus.service: Unit process 4512 (apt-get) remains running after unit stopped.
Sep  9 04:49:29 vm-02 systemd[1]: incus.service: Unit process 4590 (dpkg) remains running after unit stopped.
Sep  9 04:49:29 vm-02 systemd[1]: incus.service: Unit process 4591 (sh) remains running after unit stopped.
Sep  9 04:49:29 vm-02 systemd[1]: incus.service: Unit process 4592 (sh) remains running after unit stopped.
Sep  9 04:49:29 vm-02 systemd[1]: incus.service: Unit process 4593 (dpkg-status) remains running after unit stopped.
Sep  9 04:49:29 vm-02 systemd[1]: incus.service: Unit process 4596 (incus-base.post) remains running after unit stopped.
Sep  9 04:49:29 vm-02 systemd[1]: incus.service: Unit process 4741 (systemctl) remains running after unit stopped.
Sep  9 04:49:29 vm-02 systemd[1]: incus.service: Consumed 12.274s CPU time.
Sep  9 04:49:29 vm-02 systemd[1]: incus.service: Found left-over process 2384 (dnsmasq) in control group while starting unit. Ignoring.
Sep  9 04:49:29 vm-02 systemd[1]: incus.service: Found left-over process 4071 (incus.updater) in control group while starting unit. Ignoring.
Sep  9 04:49:29 vm-02 systemd[1]: incus.service: Found left-over process 4512 (apt-get) in control group while starting unit. Ignoring.
Sep  9 04:49:29 vm-02 systemd[1]: incus.service: Found left-over process 4590 (dpkg) in control group while starting unit. Ignoring.
Sep  9 04:49:29 vm-02 systemd[1]: incus.service: Found left-over process 4591 (sh) in control group while starting unit. Ignoring.
Sep  9 04:49:29 vm-02 systemd[1]: incus.service: Found left-over process 4592 (sh) in control group while starting unit. Ignoring.
Sep  9 04:49:29 vm-02 systemd[1]: incus.service: Found left-over process 4593 (dpkg-status) in control group while starting unit. Ignoring.
Sep  9 04:49:29 vm-02 systemd[1]: incus.service: Found left-over process 4596 (incus-base.post) in control group while starting unit. Ignoring.
Sep  9 04:49:29 vm-02 systemd[1]: incus.service: Found left-over process 4741 (systemctl) in control group while starting unit. Ignoring.
Sep  9 04:49:29 vm-02 incusd[4742]: time="2024-09-09T04:49:29Z" level=warning msg="Wait for other cluster nodes to upgrade their versions, cluster not started yet"
Sep  9 04:49:33 vm-02 kernel: [  408.959065] audit: type=1400 audit(1725857373.427:72): apparmor="STATUS" operation="profile_replace" profile="unconfined" name="incus_dnsmasq-lxdbr0_</var/lib/incus>" pid=4846 comm="apparmor_parser"
Sep  9 04:49:33 vm-02 dnsmasq[4847]: using only locally-known addresses for incus
Sep  9 04:49:33 vm-02 dnsmasq[4847]: using only locally-known addresses for incus
Sep  9 04:49:33 vm-02 dnsmasq-dhcp[4847]: read /var/lib/incus/networks/lxdbr0/dnsmasq.hosts/infra_dns-02.enp5s0
Sep  9 04:49:33 vm-02 dnsmasq-dhcp[4847]: read /var/lib/incus/networks/lxdbr0/dnsmasq.hosts/infra_dns-05.enp5s0
Sep  9 04:49:33 vm-02 dnsmasq-dhcp[4847]: read /var/lib/incus/networks/lxdbr0/dnsmasq.hosts/pki_tpki.enp5s0
Sep  9 04:49:34 vm-02 incusd[4742]: time="2024-09-09T04:49:34Z" level=error msg="Failed to register device" device=bootstrap err="Storage volume \"dns_bootstrap\" in project \"infra\" of type \"custom\" does not exist on pool \"remote\": Storage volume not found" instance=dns-02 instanceType=container project=infra
Sep  9 04:49:34 vm-02 incusd[4742]: time="2024-09-09T04:49:34Z" level=error msg="Failed to register device" device=bootstrap err="Storage volume \"dns_bootstrap\" in project \"infra\" of type \"custom\" does not exist on pool \"remote\": Storage volume not found" instance=dns-05 instanceType=container project=infra
Sep  9 04:49:34 vm-02 incusd[4742]: time="2024-09-09T04:49:34Z" level=warning msg="No active cluster event listener clients"
Sep  9 04:49:34 vm-02 incusd[4742]: time="2024-09-09T04:49:34Z" level=warning msg="Cluster member isn't responding" name=vm-01
Sep  9 04:49:34 vm-02 incusd[4742]: time="2024-09-09T04:49:34Z" level=warning msg="Failed adding member event listener client" err=EOF local="<VM2_IP>:8443" remote="<VM1_IP>:8443"
Sep  9 04:49:34 vm-02 incusd[4742]: time="2024-09-09T04:49:34Z" level=warning msg="Cluster member isn't responding" name=vm-01
Sep  9 04:49:34 vm-02 incusd[4742]: time="2024-09-09T04:49:34Z" level=warning msg="Cluster member isn't responding" name=vm-03
Sep  9 04:49:34 vm-02 incusd[4742]: time="2024-09-09T04:49:34Z" level=warning msg="Cluster member isn't responding" name=vm-01
Sep  9 04:49:34 vm-02 incusd[4742]: time="2024-09-09T04:49:34Z" level=error msg="Failed updating instance types" err="context canceled"
Sep  9 04:49:34 vm-02 incusd[4742]: time="2024-09-09T04:49:34Z" level=warning msg="Failed to update instance types: Get \"https://images.linuxcontainers.org/meta/instance-types/aws.yaml\": context canceled"
Sep  9 04:49:34 vm-02 incusd[4742]: time="2024-09-09T04:49:34Z" level=warning msg="Could not handover member's responsibilities" err="Failed to transfer leadership: No online voter found"
Sep  9 04:49:35 vm-02 incusd[4742]: time="2024-09-09T04:49:35Z" level=warning msg="Failed to resolve warning" err="Failed to resolve warnings: Failed to begin transaction: sql: database is closed"
Sep  9 04:49:35 vm-02 incusd[4742]: time="2024-09-09T04:49:35Z" level=warning msg="Failed to resolve warning" err="Failed to resolve warnings: Failed to begin transaction: sql: database is closed"
Sep  9 04:49:35 vm-02 incusd[4742]: time="2024-09-09T04:49:35Z" level=error msg="Failed updating cluster heartbeats" err="Failed to begin transaction: sql: database is closed"
Sep  9 04:49:36 vm-02 systemd[1]: incus.service: Deactivated successfully.
Sep  9 04:49:36 vm-02 systemd[1]: incus.service: Unit process 4071 (incus.updater) remains running after unit stopped.
Sep  9 04:49:36 vm-02 systemd[1]: incus.service: Unit process 4512 (apt-get) remains running after unit stopped.
Sep  9 04:49:36 vm-02 systemd[1]: incus.service: Unit process 4590 (dpkg) remains running after unit stopped.
Sep  9 04:49:36 vm-02 systemd[1]: incus.service: Unit process 4591 (sh) remains running after unit stopped.
Sep  9 04:49:36 vm-02 systemd[1]: incus.service: Unit process 4592 (sh) remains running after unit stopped.
Sep  9 04:49:36 vm-02 systemd[1]: incus.service: Unit process 4593 (dpkg-status) remains running after unit stopped.
Sep  9 04:49:36 vm-02 systemd[1]: incus.service: Unit process 4847 (dnsmasq) remains running after unit stopped.
Sep  9 04:49:36 vm-02 systemd[1]: incus.service: Unit process 4881 (incus-ui-canoni) remains running after unit stopped.
Sep  9 04:49:36 vm-02 systemd[1]: incus.service: Unit process 4884 (systemctl) remains running after unit stopped.
Sep  9 04:49:36 vm-02 systemd[1]: incus.service: Consumed 1.623s CPU time.
Sep  9 04:49:36 vm-02 systemd[1]: incus.service: Found left-over process 4071 (incus.updater) in control group while starting unit. Ignoring.
Sep  9 04:49:36 vm-02 systemd[1]: incus.service: Found left-over process 4512 (apt-get) in control group while starting unit. Ignoring.
Sep  9 04:49:36 vm-02 systemd[1]: incus.service: Found left-over process 4590 (dpkg) in control group while starting unit. Ignoring.
Sep  9 04:49:36 vm-02 systemd[1]: incus.service: Found left-over process 4591 (sh) in control group while starting unit. Ignoring.
Sep  9 04:49:36 vm-02 systemd[1]: incus.service: Found left-over process 4592 (sh) in control group while starting unit. Ignoring.
Sep  9 04:49:36 vm-02 systemd[1]: incus.service: Found left-over process 4593 (dpkg-status) in control group while starting unit. Ignoring.
Sep  9 04:49:36 vm-02 systemd[1]: incus.service: Found left-over process 4847 (dnsmasq) in control group while starting unit. Ignoring.
Sep  9 04:49:36 vm-02 systemd[1]: incus.service: Found left-over process 4881 (incus-ui-canoni) in control group while starting unit. Ignoring.
Sep  9 04:49:36 vm-02 systemd[1]: incus.service: Found left-over process 4884 (systemctl) in control group while starting unit. Ignoring.
Sep  9 04:49:40 vm-02 kernel: [  415.649770] audit: type=1400 audit(1725857380.119:73): apparmor="STATUS" operation="profile_replace" info="same as current profile, skipping" profile="unconfined" name="incus_dnsmasq-lxdbr0_</var/lib/incus>" pid=4985 comm="apparmor_parser"
Sep  9 04:49:40 vm-02 dnsmasq[4986]: using only locally-known addresses for incus
Sep  9 04:49:40 vm-02 dnsmasq[4986]: using only locally-known addresses for incus
Sep  9 04:49:40 vm-02 dnsmasq-dhcp[4986]: read /var/lib/incus/networks/lxdbr0/dnsmasq.hosts/infra_dns-02.enp5s0
Sep  9 04:49:40 vm-02 dnsmasq-dhcp[4986]: read /var/lib/incus/networks/lxdbr0/dnsmasq.hosts/infra_dns-05.enp5s0
Sep  9 04:49:40 vm-02 dnsmasq-dhcp[4986]: read /var/lib/incus/networks/lxdbr0/dnsmasq.hosts/pki_tpki.enp5s0
Sep  9 04:49:40 vm-02 incusd[4887]: time="2024-09-09T04:49:40Z" level=error msg="Failed to register device" device=bootstrap err="Storage volume \"dns_bootstrap\" in project \"infra\" of type \"custom\" does not exist on pool \"remote\": Storage volume not found" instance=dns-02 instanceType=container project=infra
Sep  9 04:49:40 vm-02 incusd[4887]: time="2024-09-09T04:49:40Z" level=error msg="Failed to register device" device=bootstrap err="Storage volume \"dns_bootstrap\" in project \"infra\" of type \"custom\" does not exist on pool \"remote\": Storage volume not found" instance=dns-05 instanceType=container project=infra
Sep  9 04:49:43 vm-02 incus.update[5085]: incus packages update has failed with code 100
Sep  9 04:49:43 vm-02 incusd[4887]: time="2024-09-09T04:49:43Z" level=warning msg="Aborting heartbeat round" err="context canceled" mode=normal
Sep  9 04:49:43 vm-02 incusd[4887]: time="2024-09-09T04:49:43Z" level=warning msg="Rejecting heartbeat request as shutting down"
Sep  9 04:49:43 vm-02 incusd[4887]: time="2024-09-09T04:49:43Z" level=error msg="Failed cleaning up operations"
Sep  9 04:49:44 vm-02 incusd[4887]: time="2024-09-09T04:49:44Z" level=warning msg="Rejecting heartbeat request as shutting down"
Sep  9 04:49:44 vm-02 systemd[1]: incus.service: Deactivated successfully.
Sep  9 04:49:44 vm-02 systemd[1]: incus.service: Unit process 4071 (incus.updater) remains running after unit stopped.
Sep  9 04:49:44 vm-02 systemd[1]: incus.service: Unit process 4986 (dnsmasq) remains running after unit stopped.
Sep  9 04:49:44 vm-02 systemd[1]: incus.service: Unit process 5086 (systemctl) remains running after unit stopped.
Sep  9 04:49:44 vm-02 systemd[1]: incus.service: Consumed 4.230s CPU time.
Sep  9 04:49:44 vm-02 systemd[1]: var-lib-incus\x2dlxcfs.mount: Deactivated successfully.
Sep  9 04:49:44 vm-02 systemd[1]: incus-lxcfs.service: Main process exited, code=exited, status=1/FAILURE
Sep  9 04:49:44 vm-02 fusermount[5116]: /bin/fusermount: failed to unmount /var/lib/incus-lxcfs: Invalid argument
Sep  9 04:49:44 vm-02 systemd[1]: incus-lxcfs.service: Failed with result 'exit-code'.
Sep  9 04:49:44 vm-02 lxcfs[5118]: Starting LXCFS at /opt/incus/bin/lxcfs
Sep  9 04:49:44 vm-02 systemd[1]: incus.service: Found left-over process 4986 (dnsmasq) in control group while starting unit. Ignoring.
Sep  9 04:49:45 vm-02 kernel: [  421.381555] audit: type=1400 audit(1725857385.851:74): apparmor="STATUS" operation="profile_replace" info="same as current profile, skipping" profile="unconfined" name="incus_dnsmasq-lxdbr0_</var/lib/incus>" pid=5219 comm="apparmor_parser"
Sep  9 04:49:45 vm-02 dnsmasq[5220]: using only locally-known addresses for incus
Sep  9 04:49:45 vm-02 dnsmasq[5220]: using only locally-known addresses for incus
Sep  9 04:49:45 vm-02 dnsmasq-dhcp[5220]: read /var/lib/incus/networks/lxdbr0/dnsmasq.hosts/infra_dns-02.enp5s0
Sep  9 04:49:45 vm-02 dnsmasq-dhcp[5220]: read /var/lib/incus/networks/lxdbr0/dnsmasq.hosts/infra_dns-05.enp5s0
Sep  9 04:49:45 vm-02 dnsmasq-dhcp[5220]: read /var/lib/incus/networks/lxdbr0/dnsmasq.hosts/pki_tpki.enp5s0
Sep  9 04:49:46 vm-02 incusd[5120]: time="2024-09-09T04:49:46Z" level=error msg="Failed to register device" device=bootstrap err="Storage volume \"dns_bootstrap\" in project \"infra\" of type \"custom\" does not exist on pool \"remote\": Storage volume not found" instance=dns-05 instanceType=container project=infra
Sep  9 04:49:46 vm-02 incusd[5120]: time="2024-09-09T04:49:46Z" level=error msg="Failed to register device" device=bootstrap err="Storage volume \"dns_bootstrap\" in project \"infra\" of type \"custom\" does not exist on pool \"remote\": Storage volume not found" instance=dns-02 instanceType=container project=infra
Sep  9 04:50:06 vm-02 incusd[5120]: time="2024-09-09T04:50:06Z" level=warning msg="Cluster member isn't responding" name=vm-01
Sep  9 04:50:06 vm-02 incusd[5120]: message repeated 3 times: [ time="2024-09-09T04:50:06Z" level=warning msg="Cluster member isn't responding" name=vm-01]

Question
Is the restarting the incus daemon twice on nodes where INCUS_CLUSTER_UPDATE script run automatically an expected behavior?

2 Likes