Introduction
Zabbly now offers custom Github Actions runners as a service on an Incus cluster in Canada.
This works very similarly to Github’s own paid runners offering but at a slightly lower price point and with a fair bit more flexibility.
Instead of using the stock Ubuntu-based Github image which has grown to be quite a massive beast with a lot of 3rd party software installed to cover all the various ecosystems, the Zabbly runners use stock Incus cloud images.
Not only does that provide a much cleaner environment, it also allows the use of distributions other than Ubuntu. The system currently supports Alma Linux, Arch, CentOS, Debian, Fedora, OpenEuler, OpenSUSE, Rocky Linux and Ubuntu.
It’s also possible to support other customer-provided images so long as they support cloud-init and are able to run the Github runner binary.
And as they’re running on top of Incus/KVM rather than Azure/Hyper-V, they’ve also generally had more reliable support for nested virtualization.
Technical details
This makes use of GARM and an Incus cluster made of 3rd gen 64-core AMD EPYC servers (similar to Github’s own setup). Integration with Github is done through a Github Application which gets enabled at the organization level and feeds in pending Github workflows back to Zabbly’s GARM instance.
Various Github runner configurations can be defined, with different resources (CPU, RAM, SSD) and images. 64-bit Arm is also available on request (limited capacity).
Drop broken fedora 44 arm64 · stgraber-test-org/test@56c16b9 · GitHub shows an example test run of all the available images on both architectures.
test/.github/workflows/runner-test.yml at main · stgraber-test-org/test · GitHub shows the Github workflow used for the test above. You effectively use the “self-hosted” tag, combined with a tag named after the runner configuration and then if dealing with multiple images and architectures, you can add those specific tags too.
Usage is tracked at a minute granularity and then automatically invoiced at the end of the month.
To avoid surprises, a spending cap can be set, once hit, no more runners will be started until the end of the month reset.
History
This infrastructure was initially put together to handle the Zabbly package builds for Incus, Linux, OVS/OVN and ZFS, then expanded to also cover incus-deploy, then IncusOS and now a variety of Incus tests too.
It was then further expanded to handle some initial customer repositories before cleaning everything up and turning into the current self-served solution with customer-managed configuration and automatic billing.