So after running a whole bunch of tests I have a better picture of what the issue is. Reproducing the issue wasn’t hard, I just had to reproduce my installation and it’s there. It took quite a long time to figure out exactly what was happening.
The main issue is related to
/ being in different mount points and the precise time in which systemd reads the
During installation, incus keeps a set of service files in
/opt/incus/lib/systemd/system and creates symlinks from the system files in
/etc/systemd/system and some subfolders. The set of files in question are
How I reproduce it
This is how I reproduced this issue using the netinstall version of Debian 12. But the issue has nothing to do with it. It’s probably reproducible on a VM from the images server. I couldn’t reproduce it there before because I had not figured out what the actual issue was.
So I created an empty incus vm, installed debian and created the following lines in fstab, where the subvolumes were created beforehand. (I tested with btrfs subvolumes, but the same issue might occur independent of the file system).
UUID=4e02e3e8-e36d-410e-8734-f9ed31d2e7fb / btrfs subvol=@,noatime,compress=zstd:1 0 0
UUID=4e02e3e8-e36d-410e-8734-f9ed31d2e7fb /opt btrfs subvol=@opt,noatime,compress=zstd:1 0 0
Now install incus and reboot the machine. Incus will not be started and the services will not be found.
To see this, just check the incus.socket service as follows.
root@debian:~# systemctl status incus.socket
Unit incus.socket could not be found.
The incus.socket file can be read and was not overwritten in any way, as the
/opt mount is there now.
root@debian:/etc/systemd/system# cat incus.socket
Description=Incus - Daemon (unix socket)
The indication that it’s a systemd loading order issue related to the mount point, we can reload systemd and see that it finds the service now, although it’s not active.
root@debian:/etc/systemd/system# systemctl daemon-reload
root@debian:/etc/systemd/system# systemctl status incus.socket
○ incus.socket - Incus - Daemon (unix socket)
Loaded: loaded (/etc/systemd/system/incus.socket; enabled; preset: enabled)
Active: inactive (dead)
Triggers: ● incus.service
Listen: /var/lib/incus/unix.socket (Stream)
This is something which I don’t know yet and still need to do some research and opinions are welcome. Searching online I did find an option for
RequiresMountsFor=, but if the service file cannot be accessed, then I’m not sure it would make a difference. Maybe just actually place the files there?
On systemd.unit manual:
Takes a space-separated list of absolute paths. Automatically adds dependencies of type Requires= and After= for all mount units required to access the specified path.
Mount points marked with noauto are not mounted automatically through local-fs.target, but are still honored for the purposes of this option, i.e. they will be pulled in by this unit.
I’m not sure the solution should be only on my side or if the incus installation should adapt a little. I will remove the mount point for /opt as a initial solution on my machines. I do know that putting
/opt in a different partition is not uncommon (I got the recommendation online when studying about snapper). But even if it is, I think there should be some notes on the installation documentation to warn users about this issue.