IncusOS first impressions on a laptop

IncusOS
1

Hi *,

I wanted to share my observations after my first installation of IncusOS on real hardware (an unused Dell XPS 9500 laptop with an i7-10750H, 32 GB RAM and 2x1TB NVMe, NVIDIA GTX 1650 Ti and only 3 USB-C ports (Ethernet via an adapter)):

  • the installation method is not compatible with Ventoy: the ISO stays stuck on “IncusOS is starting”; haven’t tried with the raw .img but not sure Ventoy knows how to deal with those
  • having two drives is a drag as it forces to boot the machine on any other Linux LiveUSB first to retrieve the correct /dev/disk/by-id/ name for the desired install/system drive so as to use it while generating the installation image
  • no issues otherwise during the install or first boot, everything works fine
  • we need a way to switch off the screen in IncusOS: besides the power saving this machine sits in my living room and I’d rather not have the display constantly on
  • It’s definitely a bit jarring when you’re not used to it not to have access to a shell to run lsblk and such (info reported by incus admin os system show resources are not the most readable, especially sizes)

That’s it so far, now I’m looking at either extending the ZFS RAID-0 pool to include the second 1 TB NVMe (this is not for production) or convert it to a RAID1. Currently reading Storage - IncusOS documentation to make sure I get it right…

But I love the idea of not having to manage the underlying server while getting the whole range of virtualization/containers options. This was much less daunting to setup than Fedora CoreOS :smile:

Thanks for the hard work and the video tutorials showing how to use/interact with this system!

2

Ventoy may be trying to access the content of the ISO which wouldn’t work as our ISO is actually a GPT formatted 2048 byte aligned image rather than ISO9660.

3

The error message should tell you about the available options, didn’t it?

4

@gibmat thoughts on that one?
Not sure what’s needed to do that, basically leave the content on screen but trigger the DPMS to shut off the screen, but then also know when to turn it back on which may be a bit tricky since we don’t support any kind of input.

I guess one option would be to implement a /1.0/system/power API which among other things would have an option to handle the DPMS stuff, either with a timeout + input support or the easy alternative of just shutting off the screen completely after x time since startup.

5

Did you check incus admin os system show storage see if that’s more usable in this scenario?

6

That was my thought too. I had to dig into my bag of old thumb drives to find one I could flash the raw img onto, but it makes sense.

7

Oh absolutely, it helpfully did! I took a picture of it with my phone and used the shortest uniquely identifying string which fortunately was quite short. It’s just having to redownload an IMG and reflash it on that old thumb drive which made me hope for a simple text menu in the installer allowing to choose which drive we should install on.

I know it breaks the unattended install aspect, but that would considerably reduce the load of having to get a custom ISO for each machine with more than one drive :smiley:

8

We currently don’t allow expanding the local pool to another device, in theory we could but that would obviously be a bit odd as we’d be mirroring a large-ish partition with a full disk.

9

Sorry, I phrased that like it’s going to be a common usage pattern while I’m definitely the odd one repurposing old laptops as servers (all my home servers are just that). I understand your target audience isn’t likely to be running IncusOS on laptops.

Still, your suggestion about a /1.0/system/power API sounds excellent. These are my notes for when I implemented that for my previous laptop-turned-home server:

Try adding consoleblank=60 or any mount of seconds that you want to the GRUB_CMDLINE_LINUX_DEFAULT= line in the /etc/default/grub file. Then run sudo update-grub and reboot the system.

From Reddit - The heart of the internet , adding video=LVDS-1:d as kernel parameter disables the screen completely (also suggested in Turn off Proxmox primary monitor. | Proxmox Support Forum ). Note however:

More recently I had a case where I wanted to access the physical device a little more frequently which made it super annoying to have to reboot to get the screen back on, so I removed those and added consoleblank=60 instead. This keeps the keyboard active and turns off the screen after 60 seconds of inactivity. So I can just hit a key to wake the screen again. I think I’m still saving the power like before and it seems to turn off the backlight.

10

Ah yeah, having to download a new image is a bit annoying in this scenario indeed.

It’s often less of an issue with servers as you can predict the prefix pretty easily and the device name supports partial matches.

1 Like
11

Yeah, we’re going to have to find a way to do it through /sys, we should be able to hit /sys/class/backlight or something like this. The kernel boot parameters in IncusOS are part of the signed data so they are completely immutable.

1 Like
12

Well once getting used to it it certainly has all the important info, but the sizes displayed as power of 10 exponents are really not usable for me (but I guess that’s a limitation of YAML):

config: {}
state:
  drives:
  - boot: false
    bus: nvme
    capacity_in_bytes: 1.024209543168e+12
    id: /dev/disk/by-id/nvme-PC611_NVMe_SK_hynix_1TB_CD07N8499102Y832V
    model_family: ""
    model_name: PC611 NVMe SK hynix 1TB
    remote: false
    removable: false
    serial_number: CD07N8499102Y832V
    smart:
      enabled: true
      passed: true
  - boot: true
    bus: nvme
    capacity_in_bytes: 1.024209543168e+12
    id: /dev/disk/by-id/nvme-PC711_NVMe_SK_hynix_1TB__KNA8N42181070865T
    model_family: ""
    model_name: PC711 NVMe SK hynix 1TB
    remote: false
    removable: false
    serial_number: KNA8N42181070865T
    smart:
      enabled: true
      passed: true
  pools:
  - devices:
    - /dev/disk/by-id/nvme-PC711_NVMe_SK_hynix_1TB__KNA8N42181070865T-part11
    encryption_key_status: available
    name: local
    pool_allocated_space_in_bytes: 4.472832e+06
    raw_pool_size_in_bytes: 9.8784247808e+11
    state: ONLINE
    type: zfs-raid0
    usable_pool_size_in_bytes: 9.8784247808e+11

But the webUI has great display for those info so no complaints here, just some getting used to.

13

Ah yeah, we need to implement some info commands that can do more than just show the raw YAML values.

1 Like
14

Ok, I was wondering about that and even wondered if I should use an external SSD to run IncusOS from and just define the storage pool later. I tried to find in the docs a way to make a custom seed marking the default storage pool to span both disks but it wasn’t immediately obvious to me (though I just skimmed the docs, I’ll dive deeper into it tonight/this weekend).

You mention mirroring, but the local pool is raid0 isn’t it? In this case it’s not mirroring two uneven drives/partitions, but just extending the existing pool (I don’t know whether ZFS allows that, I’ve had only very basic contact with it so far).

15

It ought to be easy enough to turn the screen off or on via an API. If we detect we’re running on a laptop, we could also add some logic to automatically turn off the screen after some period of inactivity.

2 Likes
16

Currently we keep the local storage completely independent, but if we relax that, we’d then support both raid1 and raid0.

In general the recommendation is to use a pretty small drive for the OS. Here I go for the higher end 256GB NVMes for that, then have other drives for the actual large storage pools.

Obviously on a laptop, it’s a bit different, but if you have two identically sized SSDs in the machine, I’d actually recommend getting a fast USB drive and put the OS on that instead. IncusOS isn’t really demanding on its OS drive, the most active thing stored on there is the Incus database and that’s just a few MBs of sqlite3 typically.

1 Like
17

If I may, as a “hobbyist”, a killer Proxmox feature is that it supports running on just a pair of drives and sets up a properly mirrored boot with everything on ZFS.

This means I can get the cheapest Hetzner iron with 2x SSDs. Getting a third drive, even a small USB flash drive increases the monthly price (1.60€/month for 16gb, guessing IncusOS wants the 64gb one which is 2.50€/month?), plus now if the USB drive dies, it’s likely annoying.

This is one of the main reason I use Proxmox instead of TrueNAS Scale on another two places too- it feels more reliable.

(Superstoked to see IncusOS released today; I’m playing with an instance right now on my Hetzner Proxmox and it looks wonderful. Even if it doesn’t support just two mirrored drives, I might use it…)

18

The systemd sysupdate partition stuff we’re using for the core of the OS itself doesn’t support RAID1 so that makes things a bit harder. Also RAID1 when dealing with UEFI ESP partitions and you want the early boot stuff TPM measured gets a bit tricky.

Basically the partition scheme looks something like this:

  • EFI ESP (2GiB)
  • seed data (100MiB)
  • A-side root partition signing (16kiB)
  • A-side root partition hashes (100MiB)
  • A-side root partition (1GiB)
  • B-side root partition signing (16KiB)
  • B-side root partition hashes (100Mib)
  • B-side root partition (1GiB
  • swap (4GiB)
  • local data (25 GiB)
  • ZFS pool (remaining space)

So we basically will consume the first 35GiB of the drive for the OS, the rest goes to ZFS. We could support RAID1 to a same size device where we effectively keep everything empty until the ZFS pool partition, leaving us some flexibility about eventual support for RAID1 on the OS stuff too.

@gibmat any thoughts on this?
Am I right that systemd-repart doesn’t support anything useful for RAID1?

And the systemd auto partition mount stuff looks at raw partitions and doesn’t know how to assemble dm devices, so doing any RAID outside of ZFS will currently be quite the PITA.

19

Yeah, no worries, I think this must be very hard because only Proxmox does it… and I think it’s not doing any of the modern stuff, including secure boot (nor A/B boot, etc.)

Looking at what Proxmox did, I see a 1mb “BIOS boot” partition, a 512Mb EFI thing, and the rest of the disk is the ZFS pool, which contains the entire root system. If I’m not mistaken, they use GRUB to boot / off ZFS. (Even swap is in the ZFS pool, which I think is problematic.) I suspect that kind of layout is not well-supported by anything and their installer is quite bespoke.

Just maybe something for the future- or to request support from underlying projects upstream.

edit: BTW; this is terribly hacky, but just in case someone comes across this… I’ve often thought of taking a Proxmox install, stripping all the Proxmox bits that provide the UI and their LXC/QEMU implementation… and just install Incus on top. It’s a shoddy Frankenstein monster, but it should do the trick…

20

Yeah, systemd-repart only deals directly with the drive. I don’t think there’s any support for a mdadm-style RAID. And it’s super hacky to try to get an ESP partition on a RAID1 device that keeps the UEFI, kernel, and mdadm all happy.

At the moment, as far as IncusOS is concerned we don’t really “care” that much about the main system drive. The expectation is that any “important” data will live on other dedicated pools, whether local or via one of the many remote storage options. That way if the system dies for whatever reason, in a high-availability setup you can just reinstall IncusOS, rejoin it to the cluster and you’re back in business. (“Treat your servers like cattle, not pets.”)

It wouldn’t be too hard to add the ability to setup mirroring of the ZFS local pool to another local drive. I wouldn’t want to support much more customization of the default local ZFS pool, just to keep the management logic simple, since it will need to run on every IncusOS system.