Extremely happy with incusOS but a bit lost

IncusOS
1

Hi there!

First of all I want to congratulate the team behind incusOS. I can’t belive it’s not more popular!

Second sorry for the LONG post

A bit of background:

I have been using incus (and lxc and lxd) in one way or another for a long time. I started with ubuntu which I keep updated but keep in a LTS release for very long before wanting to change for the next. This led to the use of lxd back then: I could have containers with newer version of the software while the base kept constant. Last year I jump ship to nixos. I though I would be wise to have an immutable system as a base and all the incus containers needed. I even started using cloud-init to create images I could then reuse (I use containers for “crash” isolation which ran podmans containers). I could rebuild the image from time to time, rebase the containers, and update the podman containers but taking also care of nixos proved to be a burden

Then incusOS came into consideration. I was bit hesitant at first but I finally managed to install it and reuse the same incus containers I had in the previous iteration of the server. I was immediatly so impressed with it I brought home a family server running proxmox which I “sometimes” updated and move it to incusOS. I could even reuse the same images I built for my system!

However, they way I see it incusOS is a very “opinated” OS. It’s not a critic, is just that I’m missing some tools from my previous OS (ubuntu, nixos) but mainly I’m missing the rationale. And I’m a little bit lost.

But I’m here for the long run, and I’m here to learn.

So the two main issues I’ve been having are: tailscale and mdadm+lvm (and smartctl)

Tailscale:

My network topology is for location #1:

  • router1: 10.10.0.0/24
  • server1: 10.10.0.10/32 with
  • incusbr0: 10.10.10.0/24 without nat

and similar for #2:

  • router2: 10.20.0.0/24
  • server2: 10.20.0.10/32 with
  • incusbr0: 10.20.10.0/24 without nat

I have tailscale installed in both routers (running openwrt) but I wanted to setup tailscale also in the server so it could use the server CPU when transfering between incusOS host servers (in different localtion) but also between incuOS containers.

I tried using the integrated tailscaled incusOS has but apparently if I add network 10.10.10.0/24 it would route everything through tailscale even when I’m local (10.10.0.0/24) because it adds a route with very high preference which cannot be overridden in incusOS

With this in mind

  • I left incusOS tailscaled enabled but without any routes (not accepting any either)
  • I create a sidecar container with tailscaled in it which I could control better
  • I added the routing to the openwrt router
  • I added dhcp option 121

It works but I don’t know if this is the correct way of doing it. Should I “ask” for a way to control the routing table in incusOS or was it deliberately designed this way? If so, is the sidecar container the correct way of doing this?

mdadm+lvm:

Due to “historical” reason I have a large (7x10TB raid6) mdadm+lvm array. I’m thinking about moving it to zfs (through the means of a cloud provider maybe?) but for the time being it is what it is.

If I understand correctly incusOS is primarily zfs (for local) so mdadm and lvm are not supported. Thus I created an incus VM (so it could have its own kernel for NFS) passthrough the drives to the container, and inside brought up mdadm array and lvm so I could share it with NFS. Then in each container I added a NFS (fuse) mount. Not as clean as “incus config edit” but it works.

I know this problem will probably go away if I decide to move to zfs raidz2 but for now. Is it the correct way of doing it?

smartclt:

Which brings as to the last question. If I want to run smartctl on the drives is passing the to a privileged container (that I would then immediatly stop) to run smarctl the correct way of doing things?

So in summary: there is no turning back to any other OS, but I sometimes feel a little bit lost and I don’t know if the things I’m doing to circumvent some of my problems are solutions or just hacks I should avoid.

Regards!

2

That’s definitely one way to do it and that’s generally fine.

IncusOS basically just runs the tailscale command and we’ve exposed a number of its common flags so far. We’re not directly interacting with the Tailscale advertised routes at all, that’s the tailscale daemon managing those. I had a quick check and didn’t see any option in the tailscale CLI to alter route priority/metric, but maybe I missed something?

3

Yeah, passing the disks to a VM is indeed the most common option to deal with other storage technologies than what’s supported natively.

Hopefully you have a way to progressively move that data onto ZFS without having to get more drives (in this economy…) or use cloud storage (in this economy…).

4

Yeah, that should work, though maybe check what’s exposed by Incus in /1.0/metrics, specifically the stuff with the node_ prefix as that’s effectively IncusOS running node-exporter and including those metrics. I wouldn’t be surprised if node-exporter actually pulled the SMART data on all drives already.

Did you also check the SMART data we expose in incus admin os system storage show?
That should show up even for drives that aren’t in a ZFS pool.

5

IncusOS is still pretty new and we’re definitely actively extending/tweaking it based on user feedback, so it’s always a good idea to bring things here and see if there’s something we can do natively or if a workaround is warranted :slight_smile:

On your first point about why it’s not more popular. That’s mostly because we’re trying for it not to be much more popular right now :wink: As with most projects, you only really get one chance at making noise and having people notice you and try it out.

We don’t feel like the experience of IncusOS is quite where we want it yet to start more actively promoting it and having a bunch of people evaluate it as a Proxmox or VMware alternative.

The main piece of the puzzle we want to address before we do more promotion is making it possible to deploy a full Incus environment. So not just Incus, but also the authentication/authorization stack, the monitoring stack, distributed storage, distributed networking and have a good way to go from one server to a cluster of a few servers.

We’re making good progress on those bits. We’ve recently landed the automatic deployment and management of Ceph. We’ve been doing a bunch of work on OpenFGA and Rauthy and we’ve got a fair bit of experience deploying the Grafana stack and load-balancer as well. We now need to really fit all that stuff together so you can grab an install image, not have to care about certificates and stuff, just install it, get a URL with an initial login/password, go there, login and use it.

6

No, it’s a very specific user case where apparently you need to edit the route tables so that tailscale doesnt route all traffic through it. This would imply an API no for tailscale but for ip routes in incusOS which is something maybe out of the scope. Anyway it works with the sidecar container!

7

Yes, it also works so no worries. I might try use a cloud provider temporarily, for a month or so, to transfer forth and back the data and convert to zfs, but again it’s working so I might never do it. I just wanted to know if using incus VM container was a good solution or there was a better one. But again, it works, speed is limited by the drives so not really an issue.

8

Yes, I’ve been having a look at the forum and truth is most of the new advancements are a little too much for me :wink: I just it at home for a couple of small servers and for my personal case it’s fully functional and mature. I couldn’t really ask for more

However I like the fact that is being active developed and I will keep looking at the forum because it’s an excellent way of learning.

And again congratulations for the excellent work you are doing!