Hello all,
recentlly i face an issue in incus which is really annoying. everytime systemd has an update it cause all my incus containers to stop and i must start them with hand. also on Vms systemd fails completly and i must force restart them so they start working again.
I even dont have an actual error or anything. it just stop after update
i am using archlinux and updating my systems with puppet and pacman
stgraber
January 8, 2025, 3:35pm
2
Can you look at the journalctl for when the update happened to see exactly what’s happening?
We’ve seen things like that on other distros due to using tools like needsrestart to try to figure out what services to restart following another package update, unfortunately this tool has a bit of a tendency to over-react, especially to systemd updates and want to restart far more than it should…
In any case, the journal output should show exactly what got stopped/restarted and hopefully why.
I’ve check my journals first. there is nothing special .
systemd[1]: Reload requested from client PID ('systemctl') (unit puppet.service)...
systemd[1]: Reloading...
systemd[1]: Reload requested from client PID ('systemctl') (unit puppet.service)...
systemd[1]: Reloading...
systemd[1]: Failed to fork off sandboxing environment for executing generators: Protocol error
systemd[1]: Configuration file /run/systemd/system/netplan-ovs-cleanup.service is marked world-inaccessible. This has no effect as configuration data is accessible via APIs without r>
systemd[1]: Reloading finished in 90115 ms.
systemd[1]: systemd-timedated.service: Deactivated successfully.
systemd[1]: systemd-hostnamed.service: Deactivated successfully.
systemd[1]: Starting Netfilter Tables...
systemd[1]: nftables.service: Deactivated successfully.
systemd[1]: Finished Netfilter Tables.
systemd[1]: Reexecution requested from client PID ('systemctl') (unit puppet.service)...
systemd[1]: Reexecuting.
then it gets reboot
this is when i do it with pacman :
sudo pacman -Syu
:: Synchronizing package databases...
core 116.3 KiB 326 KiB/s 00:00 [##########################################################################] 100%
extra 7.5 MiB 3.96 MiB/s 00:02 [##########################################################################] 100%
community is up to date
multilib 132.3 KiB 374 KiB/s 00:00 [##########################################################################] 100%
ccx is up to date
:: Starting full system upgrade...
resolving dependencies...
looking for conflicting packages...
Packages (5) libpng-1.6.45-1 systemd-257.2-1 systemd-libs-257.2-1 systemd-sysvcompat-257.2-1 unixodbc-2.3.12-3
Total Download Size: 10.47 MiB
Total Installed Size: 38.17 MiB
Net Upgrade Size: 0.01 MiB
:: Proceed with installation? [Y/n]
:: Retrieving packages...
systemd-257.2-1-x86_64 8.8 MiB 675 MiB/s 00:00 [##########################################################################] 100%
systemd-libs-257.2-1-x86_64 1227.5 KiB 1199 MiB/s 00:00 [##########################################################################] 100%
unixodbc-2.3.12-3-x86_64 254.5 KiB 249 MiB/s 00:00 [##########################################################################] 100%
libpng-1.6.45-1-x86_64 251.1 KiB 245 MiB/s 00:00 [##########################################################################] 100%
systemd-sysvcompat-257.2-1-x86_64 6.1 KiB 2036 KiB/s 00:00 [##########################################################################] 100%
Total (5/5) 10.5 MiB 12.6 MiB/s 00:01 [##########################################################################] 100%
(5/5) checking keys in keyring [##########################################################################] 100%
(5/5) checking package integrity [##########################################################################] 100%
(5/5) loading package files [##########################################################################] 100%
(5/5) checking for file conflicts [##########################################################################] 100%
(5/5) checking available disk space [##########################################################################] 100%
:: Processing package changes...
(1/5) upgrading libpng [##########################################################################] 100%
(2/5) upgrading systemd-libs [##########################################################################] 100%
(3/5) upgrading systemd
container stop here.
systemd[1]: Reexecution requested from client PID 181754 ('systemctl')...
systemd[1]: Reexecuting.
stgraber
January 8, 2025, 7:51pm
4
Just to be clear, the systemd update is happening inside the container or on the host?
stgraber
January 8, 2025, 7:53pm
6
Interesting. Can you try incus console --show-log NAME for a container as it gets stuck/dies on the update?
I downgrade systemd and tried to do what you asked . this is what i got :
ystemd 257.2-1-arch running in system mode (+PAM +AUDIT -SELINUX -APPARMOR -IMA +IPE +SMACK +SECCOMP +GCRYPT +GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 -IDN +IPTC +KMOD +LIBCRYPTSETUP +LIBCRYPTSETUP_PLUGINS +LIBFDISK +PCRE2 +PWQUALITY +P11KIT +QRENCODE +TPM2 +BZIP2 +LZ4 +XZ +ZLIB +ZSTD +BPF_FRAMEWORK +BTF +XKBCOMMON +UTMP -SYSVINIT +LIBARCHIVE)
Detected virtualization lxc.
Detected architecture x86-64.
bpf-restrict-fs: Failed to create BPF map: Operation not permitted
Failed to fork off sandboxing environment for executing generators: Protocol error
[!!!!!!] Failed to start up manager.
Exiting PID 1...
stgraber
January 8, 2025, 11:25pm
8
The BPF part shouldn’t be fatal, but the Failed to fork off sandboxing sounds like the culprit here.
Now having systemd straight up exit on a failed reload seems rather rude…
@Foxboron did you ever see something like this? (user is on Arch)
is there any chance that someone works on it? we had another systemd update last week and same thing happend