Incus 1.16 allows tmpfs and tmpfs-overlay disks for containers. But I am not able to use it for rootfs.
$ incus config device override <container-name> root source=tmpfs:
Error: Invalid devices: Device validation failed for "root": Root disk entry may not have a "source" property set
neither
$ incus config device add <container-name> rootoverlay disk source=tmpfs-overlay: path=/
Error: Invalid devices: Device validation failed for "rootoverlay": Root disk entry may not have a "source" property set
I found a related thread Override root device for VMs? but does not actually help. It seems that changing rootfs is not supported by design?
At this point I want to know if there is a workaround? In specific, I am trying to make a “nixos-container”(based on systemd-nspawn) counterpart that based on incus. Container will share host’s nix store, and it does not rely on rootfs like other distro. So a tmpfs as root will totally be ok. I found manually executing incus rebuild <container-name> --empty works similarly, but is there a more elegant way?
We don’t support using a tmpfs as the root disk as that would yield a completely empty / which would then lack even basic entries needed for initial container setup (/proc, /sys, /dev) and wouldn’t have a usable init binary as /sbin/init.
We also don’t currently support setting up the root disk as a tmpfs+overlay. That one would be technically doable on our end but it would need quite a bit of care in ordering all the mount entries correctly for this to work without interfering with basic mount entries (/dev, /proc, /sys) or additional mount entries coming from disk devices.
The rebuild --empty will work to clear everything, though you’ll again need something to populate a functional /sbin/init for the container to boot up.
If dealing with a limited number of paths, you could specifically make those tmpfs: or tmpfs-overlay: instead, leaving the root disk for actual persistent data (if any).
Thanks, I will probably resort to rebuild --empty.
BTW, could those folders (/proc, /sys, /dev) get created on the fly? As for /sbin/init, I use raw.lxc: lxc.init.cmd=... to point to a different location in nix store, witch is another disk device mounted separately. So don’t worry about that.
I “think” we’re tagging all of those for auto-creation but that may still mess with logic running prior to container startup (LXC/LXCFS startup hooks come to mind).