Which folders should be set on different subvolumes for an incus installation and containers to not be affected by a snapper rollback?
In other words, what are all the folders which incus uses?
Any other consideration I should have when running incus and snapper together?
You’d probably want /var/lib/incus/ itself as that contains the database amongst other things which you probably don’t want to see rolled back.
The instances themselves are usually on a storage pool which is already its own block device or subvolume so those should be okay.
You could also include /var/log/incus/ if you wanted, but that’s a lot less important.
/var/log is already in another subvolume and includes the incus log, so that is fine.
Are the images and backups which are not set in a pool by the storage.images_volume and storage.backups_volume keys also stored inside /var/lib/incus?
Those two default to /var/lib/incus/images and /var/lib/incus/backups so they would be covered.
This is a somewhat related issue, so I’ll post it here.
I mounted a btrfs subvolume on /var/lib/incus and installed incus, ran init while I was root, added my user to incus-admin and restarted the machine. After reboot, all commands that interact with the daemon return the following message: Error: Failed to connect to local daemon: Get "http://unix.socket/1.0": dial unix /var/lib/incus/unix.socket: connect: connection refused.
I think this might be an issue with the fact that, to mount a subvolume on /var/lib/incus, I had to create that directory beforehand. Right now its permissions are like this: drwx--x--x 1 root root 394 dez 26 09:48 incus. I’m wondering this is not correct. Can anyone who has a proper running installation please show me the permissions owner and group of this directory?
Btw, if this is incorrect, for the case in which that directory is already created does it make sense for incus to check and set the permissions on that directory? Or does it make more sense for the user who is creating the directory to set it up properly?
Actually, that is not the issue. Just checked on the “Try it online” version and the permissions are the same.
Running sudo systemctl enable incus --now solved it. I’m wondering why the installation did not enable it by default.
$ sudo systemctl enable incus --now
Created symlink /etc/systemd/system/incus.service → /opt/incus/lib/systemd/system/incus.service.
Failed to enable auxiliary unit incus-startup.service, ignoring.
Failed to enable auxiliary unit incus.socket, ignoring.
The unit files have no installation config (WantedBy=, RequiredBy=, Also=,
Alias= settings in the [Install] section, and DefaultInstance= for template
units). This means they are not meant to be enabled using systemctl.
Possible reasons for having this kind of units are:
• A unit may be statically enabled by being symlinked from another unit's
.wants/, .requires/, or .upholds/ directory.
• A unit's purpose may be to act as a helper for some other unit which has
a requirement dependency on it.
• A unit may be started when needed via activation (socket, path, timer,
D-Bus, udev, scripted systemctl call, ...).
• In case of template units, the unit is meant to be enabled with some
instance name specified.
$ incus list
+------------+---------+--------------------+-----------------------------------------------+-----------+-----------+
| NAME | STATE | IPV4 | IPV6 | TYPE | SNAPSHOTS |
+------------+---------+--------------------+-----------------------------------------------+-----------+-----------+
| controller | RUNNING | 192.168.0.7 (eth0) | 2804:29b8:50f4:2476:216:3eff:fe0f:f749 (eth0) | CONTAINER | 0 |
+------------+---------+--------------------+-----------------------------------------------+-----------+-----------+
That’s odd, the daemon isn’t meant to start automatically but instead be socket activated through the incus.socket unit, this causes any connection to /var/lib/incus/unix.socket from a user that’s part of the incus-admin group to kick the incus.service unit.
If that doesn’t happen, it’d be good to check the status of the incus.socket unit.
This is after I have enabled incus service. Everything is running fine. Last time I rebooted, I had to enable the incus service again.
victoitor@escritorio:~$ sudo systemctl | grep incus
var-lib-incus-guestapi.mount loaded active mounted /var/lib/incus/guestapi
var-lib-incus-shmounts.mount loaded active mounted /var/lib/incus/shmounts
var-lib-incus-storage\x2dpools-default.mount loaded active mounted /var/lib/incus/storage-pools/default
var-lib-incus.mount loaded active mounted /var/lib/incus
var-lib-incus\x2dlxcfs.mount loaded active mounted /var/lib/incus-lxcfs
incus-lxcfs.service loaded active running Incus - LXCFS daemon
incus.service loaded active running Incus - Daemon
incus.socket loaded active running Incus - Daemon (unix socket)
victoitor@escritorio:~$ sudo systemctl status incus.socket
â—Ź incus.socket - Incus - Daemon (unix socket)
Loaded: loaded (/etc/systemd/system/incus.socket; enabled; preset: enabled)
Active: active (running) since Tue 2023-12-26 20:16:21 -03; 1h 23min ago
Triggers: â—Ź incus.service
Listen: /var/lib/incus/unix.socket (Stream)
Tasks: 0 (limit: 18841)
Memory: 0B
CPU: 1ms
CGroup: /system.slice/incus.socket
dez 26 20:16:20 escritorio systemd[1]: Starting incus.socket - Incus - Daemon (unix socket)...
dez 26 20:16:21 escritorio systemd[1]: Listening on incus.socket - Incus - Daemon (unix socket).
Going to reatsrt the machine and see if the same issue happens again.
EDIT:
After every reboot the problem comes back. It’s interesting this issue is happening in two machines, but I have tried to reproduce it on an incus vm and couldn’t. Rebooting also hangs for about a minute waiting for incusd and systemd services.
This is what happens after a reboot:
victoitor@escritorio:~$ incus list
Error: The incus daemon doesn't appear to be started (socket path: /var/lib/incus/unix.socket)
victoitor@escritorio:~$ sudo systemctl | grep incus
var-lib-incus.mount loaded active mounted /var/lib/incus
victoitor@escritorio:~$ sudo systemctl status incus.socket
Unit incus.socket could not be found.
victoitor@escritorio:~$ sudo systemctl enable incus --now
Failed to enable auxiliary unit incus-startup.service, ignoring.
Failed to enable auxiliary unit incus.socket, ignoring.
The unit files have no installation config (WantedBy=, RequiredBy=, Also=,
Alias= settings in the [Install] section, and DefaultInstance= for template
units). This means they are not meant to be enabled using systemctl.
Possible reasons for having this kind of units are:
• A unit may be statically enabled by being symlinked from another unit's
.wants/, .requires/, or .upholds/ directory.
• A unit's purpose may be to act as a helper for some other unit which has
a requirement dependency on it.
• A unit may be started when needed via activation (socket, path, timer,
D-Bus, udev, scripted systemctl call, ...).
• In case of template units, the unit is meant to be enabled with some
instance name specified.
victoitor@escritorio:~$ incus list
+---------+---------+-------------------+------+-----------+-----------+
| NAME | STATE | IPV4 | IPV6 | TYPE | SNAPSHOTS |
+---------+---------+-------------------+------+-----------+-----------+
| ansible | RUNNING | 10.0.0.230 (eth0) | | CONTAINER | 0 |
+---------+---------+-------------------+------+-----------+-----------+
victoitor@escritorio:~$ sudo systemctl | grep incus
var-lib-incus-guestapi.mount loaded active mounted /var/lib/incus/guestapi
var-lib-incus-shmounts.mount loaded active mounted /var/lib/incus/shmounts
var-lib-incus-storage\x2dpools-default.mount loaded active mounted /var/lib/incus/storage-pools/default
var-lib-incus.mount loaded active mounted /var/lib/incus
var-lib-incus\x2dlxcfs.mount loaded active mounted /var/lib/incus-lxcfs
incus-lxcfs.service loaded active running Incus - LXCFS daemon
incus.service loaded active running Incus - Daemon
incus.socket loaded active running Incus - Daemon (unix socket)
victoitor@escritorio:~$ sudo systemctl status incus.socket
â—Ź incus.socket - Incus - Daemon (unix socket)
Loaded: loaded (/etc/systemd/system/incus.socket; enabled; preset: enabled)
Active: active (running) since Tue 2023-12-26 21:58:54 -03; 16s ago
Triggers: â—Ź incus.service
Listen: /var/lib/incus/unix.socket (Stream)
Tasks: 0 (limit: 18841)
Memory: 0B
CPU: 689us
CGroup: /system.slice/incus.socket
dez 26 21:58:54 escritorio systemd[1]: Starting incus.socket - Incus - Daemon (unix socket)...
dez 26 21:58:54 escritorio systemd[1]: Listening on incus.socket - Incus - Daemon (unix socket).
Ok, so I think the problem is with your mount.
incus.socket starts properly on boot but that happens early as part of the sockets target, then your mount on /var/lib/incus happens, hiding that unix socket.
Then when something tries to connect and trigger incus, it fails because the unix socket isn’t responding.
I suspect that if you arrange to have systemctl restart incus.socket happen after /var/lib/incus is mounted, things will work properly.
I’m not particularly sure that’s the issue. I’m guessing it’s something related to a Debian 12 netinstall (which both of my systems with this issue are). I was trying to figure out the service startup order using systemd-analyze plot and noticed basically nothing related to incus appears there.
On both my machines, I basically get just the mount point.
victoitor@escritorio:/etc/systemd/system$ systemd-analyze plot | grep incus
<text class="left" x="420.764" y="2214.000">var-lib-incus.mount (13ms)</text>
Tried checking the same on the try it online version of incus.
ubuntu@tryit:~$ systemd-analyze plot | grep incus
<text class="left" x="354.567" y="614.000">incus-agent.service (405ms)</text>
<text class="left" x="359.547" y="854.000">run-incus_agent.mount</text>
<text class="left" x="434.377" y="1274.000">dev-virtio\x2dports-org.linuxcontainers.incus.device</text>
<text class="left" x="445.557" y="2014.000">dev-disk-by\x2did-scsi\x2d0QEMU_QEMU_HARDDISK_incus_root.device</text>
<text class="left" x="455.365" y="2874.000">dev-disk-by\x2did-scsi\x2d0QEMU_QEMU_HARDDISK_incus_root\x2dpart2.device</text>
<text class="left" x="456.506" y="2934.000">dev-disk-by\x2did-scsi\x2d0QEMU_QEMU_HARDDISK_incus_root\x2dpart1.device</text>
<text class="left" x="514.698" y="3654.000">incus-user.socket (21ms)</text>
<text class="left" x="516.690" y="3674.000">incus.socket (2ms)</text>
<text class="left" x="518.664" y="3794.000">incus-growpart.service (279ms)</text>
<text class="left" x="518.917" y="3814.000">incus-lxcfs.service (18ms)</text>
<text class="left" x="519.482" y="3834.000">incus-startup.service (5.715s)</text>
<text class="left" x="527.562" y="3994.000">var-lib-incus\x2dlxcfs.mount</text>
<text class="left" x="577.511" y="4134.000">incus.service (5.642s)</text>
<text class="right" x="641.410" y="4154.000">var-lib-incus-shmounts.mount</text>
<text class="right" x="641.699" y="4174.000">var-lib-incus-guestapi.mount</text>
<text class="right" x="995.153" y="4194.000">sys-subsystem-net-devices-incusbr0.device</text>
<text class="right" x="995.155" y="4214.000">sys-devices-virtual-net-incusbr0.device</text>
There are actual services starting up here. It might not be an order thing. Btw, I tested this with the Debian 12 VM image from the image server and it starts up fine. It might be something related to the netinstall version. I know there are some changes between them. Looking at my previous post, mwhen I enable the incus service, I do get an error that it’s not possible to enable incus.socket, so it is not enabled, and that’s probably why it doesn’t start.
That’s odd because I actually reinstalled my laptop a couple of days ago using Debian 12 netinst and Incus is behaving fine here without needing any systemd tricks.
It’s good to know, so it’ll be something I don’t need to check.
I plan on running a few virtual machines with my setup to try and figure out what’s going on. I ran some experiments already but couldn’t reproduce it. It’s quite weird that it happens on both my machines. Sadly, I currently don’t have the time to do this, so I’ll report back only when I can.
It might be my partitioning scheme. Might be some interaction with what I have installed or a combination of these, but something is definitely off.
By the way, just for comparison, I’m running a few btrfs mountpoints and zram. The mountpoints are the following:
# <file system> <mount point> <type> <options> <dump> <pass>
# / was on /dev/sda2 during installation
/ btrfs noatime,compress=zstd:1,subvol=@ 0 0
/home btrfs noatime,compress=zstd:1,subvol=@home 0 0
/root btrfs noatime,compress=zstd:1,subvol=@root 0 0
/var/log btrfs noatime,compress=zstd:1,subvol=@log 0 0
/var/lib/AccountsService btrfs noatime,compress=zstd:1,subvol=@AccountsService 0 0
/var/lib/gdm3 btrfs noatime,compress=zstd:1,subvol=@gdm 0 0
/tmp btrfs noatime,compress=zstd:1,subvol=@tmp 0 0
/opt btrfs noatime,compress=zstd:1,subvol=@opt 0 0
/var/lib/incus btrfs noatime,compress=zstd:1,subvol=@incus 0 0
On the server is basically the same without the AccountsService and gdm mountpoints and the same problem exists.
So after running a whole bunch of tests I have a better picture of what the issue is. Reproducing the issue wasn’t hard, I just had to reproduce my installation and it’s there. It took quite a long time to figure out exactly what was happening.
The main issue is related to /opt and / being in different mount points and the precise time in which systemd reads the /etc/systemd/system/incus.socket file.
During installation, incus keeps a set of service files in /opt/incus/lib/systemd/system and creates symlinks from the system files in /etc/systemd/system and some subfolders. The set of files in question are incus-lxcfs.service, incus.socket, incus-user.socket and incus-startup.service.
This is how I reproduced this issue using the netinstall version of Debian 12. But the issue has nothing to do with it. It’s probably reproducible on a VM from the images server. I couldn’t reproduce it there before because I had not figured out what the actual issue was.
So I created an empty incus vm, installed debian and created the following lines in fstab, where the subvolumes were created beforehand. (I tested with btrfs subvolumes, but the same issue might occur independent of the file system).
UUID=4e02e3e8-e36d-410e-8734-f9ed31d2e7fb / btrfs subvol=@,noatime,compress=zstd:1 0 0
UUID=4e02e3e8-e36d-410e-8734-f9ed31d2e7fb /opt btrfs subvol=@opt,noatime,compress=zstd:1 0 0
Now install incus and reboot the machine. Incus will not be started and the services will not be found.
To see this, just check the incus.socket service as follows.
root@debian:~# systemctl status incus.socket
Unit incus.socket could not be found.
The incus.socket file can be read and was not overwritten in any way, as the /opt mount is there now.
root@debian:/etc/systemd/system# cat incus.socket
[Unit]
Description=Incus - Daemon (unix socket)
[Socket]
ListenStream=/var/lib/incus/unix.socket
SocketGroup=incus-admin
SocketMode=0660
Service=incus.service
[Install]
WantedBy=sockets.target
The indication that it’s a systemd loading order issue related to the mount point, we can reload systemd and see that it finds the service now, although it’s not active.
root@debian:/etc/systemd/system# systemctl daemon-reload
root@debian:/etc/systemd/system# systemctl status incus.socket
â—‹ incus.socket - Incus - Daemon (unix socket)
Loaded: loaded (/etc/systemd/system/incus.socket; enabled; preset: enabled)
Active: inactive (dead)
Triggers: â—Ź incus.service
Listen: /var/lib/incus/unix.socket (Stream)
This is something which I don’t know yet and still need to do some research and opinions are welcome. Searching online I did find an option for RequiresMountsFor=, but if the service file cannot be accessed, then I’m not sure it would make a difference. Maybe just actually place the files there?
RequiresMountsFor=
Takes a space-separated list of absolute paths. Automatically adds dependencies of type Requires= and After= for all mount units required to access the specified path.
Mount points marked with noauto are not mounted automatically through local-fs.target, but are still honored for the purposes of this option, i.e. they will be pulled in by this unit.
I’m not sure the solution should be only on my side or if the incus installation should adapt a little. I will remove the mount point for /opt as a initial solution on my machines. I do know that putting /opt in a different partition is not uncommon (I got the recommendation online when studying about snapper). But even if it is, I think there should be some notes on the installation documentation to warn users about this issue.
So I guess a systemctl daemon-reload following /opt being mounted then causes the units to properly appear?
You may need a small unit which starts after /opt is mounted and basically calls systemctl daemon-reload as well as systemctl start incus.socket incus-user.socket incus-lxcfs.service incus-startup.service to get things back up.
I find that solution to be quite convoluted. I’m not sure it makes that much sense to keep /opt in a separate partition, so I’ll remove it (and move the files). Seems simpler for now.
I still think there should be some note on this issue in the incus installation guide. It’s not easy to figure out if it happens to anybody else.
Thinking about the best solution to this problem, I think it makes sense for systemd to check if the service file itself is a symlink and wait for the directory to be mounted before trying to load the file. So reporting the issue to systemd might be the proper solution.
Does this make sense?
They’ll probably argue that the unit file ought to be in /etc/systemd or /lib/systemd instead and then have that define that it needs a particular mountpoint to be up (/opt in this case).
But it’s probably still worth the shot to report it and see if they’d be open to adding some smarts around symlinks.
In our case, as those packages aren’t proper distro packages but more vendor-type packages, I like the idea of effectively everything being in /opt/incus/ with the few bits of system-wide integration being symlinks, trying to keep things as separated as possible.
It makes sense on the incus installation.
I’m not on the computer to check now, but I remember unattended-upgrades service was also a symlink on the Debian installation. Don’t remember to where though. It might be used in the argument since it locks the user from partitioning as needed if it crosses a symlink. The unattended-upgrades in Debian is an indication incus is not the only one doing it.