Some questions and feedback from an IncusOS-curious

IncusOS
1

Hello, I discovered Incus a few days ago due to Proxmox Community Scripts gaining support for it, and I am now seriously considering straight-up replacing Proxmox! I installed IncusOS in a VM and have been messing with it, and it solves a lot of my Proxmox grievances and the OCI container support is very nice!

However, there are a few things which are preventing me from pulling the trigger for using IncusOS as my primary server (compute + NAS):

I would like finer control over ZFS; I currently have a 5x16TB RAIDZ2 (48TB usable) with mirrored 1 TB SSDs as metadata specials, and I make use of special_small_blocks to have small files and entire certain datasets live on the SSDs. I also have plans to fail out one of these NVMe drives and replace it with cheap rust, making use of the non-rotating bias to keep the NVMe responsiveness while keeping the redundancy and freeing up one of the drives for another project. All of this will require me to be able to edit all ZFS dataset and pool properties on IncusOS.

I have not yet found a way to create nested ZFS datasets. I have the following topology:

NAME                                 SPECIAL_SMALL_BLOCKS  RECSIZE   USED  AVAIL
origin                                               256K       4M  28.3T  14.7T
origin/app                                           256K       4M   276G  14.7T
origin/app/immich                                    256K       4M   276G  14.7T
origin/app/immich/thumbs                               4M       4M  9.41G  14.7T
origin/app/proxmox-backup-server                     256K       2M   356G  14.7T
origin/media                                         256K       4M  23.1T  14.7T
origin/user                                          256K       4M  4.60T  14.7T
origin/user/<name1>                                  256K       4M  61.5G   962G
origin/user/<name2>                                  256K       4M  4.53T  14.7T
origin/user/<name3>                                  256K       4M  9.88G  1014G
origin/user/public                                   256K       4M  1.48G  1023G

(most apps have their data in proxmox LXC block devices, which are not stored in this pool)

While I could get away without nesting, it makes one particular situation more annoying, and that is the immich thumbnails - I have those set up to live entirely on the SSD, and you can’t relocate the thumbnails directory independently of the rest of the upload directory, so I would have to symlink it.

I have personally switched to copyparty for my file server needs, and this can work well on Incus, but I also have some users who still prefer SMB. I use Linux users to manage SMB permissions, but IncusOS doesn’t appear to give me any user management tools.

I absolutely need Network UPS Tools support in the OS so that it can shut itself down gracefully before the power is cut.

Also, please for the love of all that is good, let me use a password for WebUI authentication. I use a password manager (KeePassXC) with very long randomly generated passwords for everything, and I do not have OIDC set up, and using client certificates doesn’t blend well with a reverse proxy.

I apologize that this was all me demanding things, so here are a few compliments:

  • The UI is pretty intuitive to use and is very snappy!
  • Having OCI containers, LXC containers, and VMs all living happily together is absolutely phenomenal and is one of the main reasons I am considering the switch (I am tired of relying on Proxmox Community Scripts to get “application containers” on Proxmox).
  • I really appreciate that you can use Incus on existing Linux servers as well as the appliance IncusOS - if not all of my needs can be met with IncusOS then I could always install Debian instead (but then I miss out on the automagic TPM management and other nice things about IncusOS).
  • Integrating natively with ZFS snapshots instead of using a bespoke system like Proxmox is very very nice. You can see in my pool layout that Proxmox Backup Server is using 356 GB, which would not be necessary if it used ZFS snapshots!

Thanks for reading my wall of text! Some of it might be misinformed or I may have missed a few things, and if so I apologize. I am super excited about Incus and hope that it will be able to fulfill all of my homelabbing needs!

2

So a few things:

  • Not sure if helpful in this situation, but Incus supports ZFS delegation so you can use the zfs tool inside of Incus containers to create additional datasets either on the main root disk or on any attached custom volumes.
  • IncusOS does support the special vdev and has support for setting the special_small_blocks_size_in_kb, see examples in Storage - IncusOS documentation
  • I’m not sure I follow on why IncusOS would need to manage users for an SMB server running inside of an Incus container. You’d just manage your users/groups inside of that container, the host doesn’t have to be involved.
  • For NUT, the best option is to run it in a container and have it trigger the shutdown through the IncusOS API. That avoids running it and all its plugins directly on the host and that trigger the correct shutdown sequence through IncusOS, allowing it to correctly shutdown its services and applications in the correct order.
  • For login, Incus has no intention on ever getting login/password type authentication. But the plan for IncusOS is to have a deployment method which bundles an OIDC IdP, providing the regular login experience that way while also providing SSO for the other support services. See Offer a full Incus deployment as part of Incus initialization · Issue #497 · lxc/incus-os · GitHub
3

Thank you for the speedy response!

  • I had no idea that “ZFS delegation” was possible, I will look into it! It does feel like a hack, but if it works, it works.
  • Ah, I swore I had seen it somewhere, but searching special_small_blocks on the docs website came up with nothing, so I assumed I had misremembered. The ZFS delegation container will let me do whatever I want regardless, so that’s no longer a concern.
  • You have a very great point about the SMB users and I apologize for not seeing that very obvious solution. :sweat_smile:
  • NUT definitely feels like it should be a system component, but I have to keep in mind the power of Incus’s CLI and remote connection capabilities. And the word “plugins” gives insight into the kinds of potential attack surfaces such a system component could introduce.
  • At the end of the day, as long as I can access the WebUI through my reverse proxy and log in with a username+password like I do for literally everything else, I’ll be happy. I probably should look into setting up a personal IdP, but it’s another item on the mile-long list of side projects, and a rather complex one at that.
4

If you just want to access the UI from incus client you can also use:

incus webui