For the first time here, I have a topic with solution instead of questions.
We have Jenkins slaves using LxD as VM of building OEM images/iso. After comparing with those build upon Vagrant VM, the build time under the same conditions is much longer. Investigation shows the bottleneck on io when getting packages for installing.
It comes out there is a property in zfs “sync” to be default set to standard.
$zfs get sync
......
local/virtual-machines/oem-iot-focal-2 sync standard default
local/virtual-machines/oem-iot-jammy-1 sync standard default
This following set property resolve the issues.
$sudo zfs set sync=disabled local/virtual-machines
$zfs get sync
.......
local/virtual-machines/oem-iot-focal-2 sync disabled inherited from local/virtual-machines
local/virtual-machines/oem-iot-jammy-1 sync disabled inherited from local/virtual-machines
The total build time is originally 9hrs, in my case, down to 50mins. It is now compare to Vagrant VM even.
Yes, but since we use LxD vm as a build slave, everything on that vm after we have the artifacts, iso/images, is not important.
Not to safe we will destroy the vm and create a new one right after every build.
IO still the bottleneck then I search and found this paragraph
While this backend is fully functional, it’s also much slower than all the others due to it having to unpack images or do instant copies of instances, snapshots and images.
Now maybe I have to switch back to zfs with sync disabled, to have kept flexibility? From the result this is the best closed to my benchmark using Vagrant VM on bare metal.
Yeah, unfortunately there aren’t any magic storage backend that’s good at everything…
For VMs stored on dir or btrfs, they’re stored in a loop file on disk, so performance isn’t going to be particularly great. ZFS is a bit better as its zvol at least offer a bit more flexibility in configuration.
The fastest for VMs I suspect would be LVM in a non-thinpool configuration, but that’s very costly in space and isn’t a great backend to run containers on, so if running both containers and VMs, that wouldn’t be so great.
Most often, I find ZFS to be the best compromise. It’s not the best at anything really, but it offers enough knobs that you can get close and still use a single backend for everything.