Easy way to try Incus on macOS with Colima

Incus support has been added to Colima, a tool that provides easy access to container runtimes on macOS.

Feedbacks would be appreciated.

Getting started with Incus on macOS with Colima

Install dependencies

# install incus client
brew install incus

# install Colima
brew install --head colima 

Start Colima using Incus runtime

colima start --runtime=incus

Start using Incus

$ incus version
Client version: 6.3
Server version: 6.3

$ incus list 
+------+-------+------+------+------+-----------+
| NAME | STATE | IPV4 | IPV6 | TYPE | SNAPSHOTS |
+------+-------+------+------+------+-----------+

$ incus info --resources | head -n5
System:
  Vendor: QEMU
  Product: QEMU Virtual Machine
  Version: virt-9.0
  Type: virtual-machine

It feels native, how does it work?

  • Colima starts an Ubuntu VM with Incus pre-installed using QEMU (or macOS Virtualization.Framework).
  • Incus unix socket is forwarded to the macOS host
  • Incus client (on macOS) is configured to utilise the forwarded unix socket. Automatically configured and removed on Colima startup and shutdown.

Other features

  • Incus UI available at https://localhost:8443
  • Volume mounts - $HOME is mounted into the Incus VM
  • Port forwarding - all ports exposed to the VM is forwarded to macOS. e.g. a proxy device listening on 8080 is accessible on localhost:8080 on macOS.
  • Docker remote is pre-configured for OCI images. e.g. incus launch docker:caddy should just work.

Limitations

  • Only containers are supported, no virtual machines.
  • Containers are not accessible on the host via the bridge IP address, unless it is further exposed to the host e.g. via proxy device.
6 Likes

Great to see!

1 Like

While testing I realised that it is possible to run x86_64 binaries utilising either QEMU or Rosetta (even in a nested container i.e. Docker in Incus)

root@aca2def12571:/# ps faux
USER         PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root           1  1.8  0.2 179432  5352 pts/0    Ss   10:01   0:00 /mnt/lima-rosetta/rosetta /bin/bash
root           9 83.3  0.2 182476  4608 pts/0    R+   10:01   0:00 /usr/bin/ps faux

Nested Virtualization is now supported on m3 Macs running macOS 15 or later.
Colima may soon be able to support virtual machines for Incus runtime.

This is Incus running on a Debian VM on macOS, able to launch both containers and virtual machines.

2 Likes

Nice! That’s great to see, will make it possible to use all of Incus on MacOS then!

How’s the performance for those VMs? Is there any known amount of overhead on the nested virtualization?

Looks like performance is impressive and the overhead is minimal.

Geekbench kept freezing in the multi-core tests, so I had to limit the VM to single CPU to get it to complete successfully. I am mainly interested in the single-core performance anyways.

You can suggest other tests or benchmarking tools to use.

The left window is the VM, right is the nested VM.

Link to the VM result: Apple Inc. Apple Virtualization Generic Platform - Geekbench
Link to the nested VM result: QEMU KVM Virtual Machine - Geekbench

2 Likes

After having tested it successfully, PRs have been created in upstream projects.

When they both get merged, Incus virtual machines would be supported in Colima.

2 Likes

Great to see! Very impressive indeed!

1 Like

I got Incus and Colima up and running pretty quickly with your documentation - thanks!

Mac Mini M1 uname -a produces;
Darwin Kernel Version 23.6.0: Mon Jul 29 21:14:21 PDT 2024; root:xnu-10063.141.2~1/RELEASE_ARM64_T8103 arm64

$ incus version
Client version: 6.5
Server version: 6.4

Kind of weird that there is a different kernel in the container and also weird that it is an Ubuntu labled kernel on Debian. Is that the ‘cloud-init’ component? Or what is going on here?
Running uname -a inside the container produces;

Linux Trixie 6.8.0-39-generic #39-Ubuntu SMP PREEMPT_DYNAMIC Sat Jul  6 02:50:39 UTC 2024 aarch64 GNU/Linux
Mac-Mini> $ incus exec Trixie -- /bin/bash
root@Trixie:~# 

Can you explain what is weird? Incus is not running directly on macOS, rather via a Linux VM.

When at that shell, what does

head /etc/lsb-release /etc/os-release /etc/debian_version

show?

It’s strange that the Incus Client and the Incus Server versions are different. They should be the same. Not critical, but this one is weird.

As already said, Incus requires a Linux system to work, and in the case of macOS (and Windows), Incus launches a Linux VM. The Incus Client is running on macOS but the Incus Server is inside the Linux VM.

When you launch Incus containers and VMs, those run inside the LinuxVM. When you launch an Incus VM, then this VM runs inside the Linux VM (nested virtualization).

2 Likes

Yeah, the client and server are totally detached and actually installed separately. It is also evident from the installation instructions.

The slight delay in version parity is due to the fact that I assume I’m pretty much the only user. I am personally always on the bleeding edge and I only push out releases at my convenience.

Going forward, I will try to push out updates shortly after new incus releases.

3 Likes

I work a lot on our Incus terraform provider with my Mac and used Orbstack’s systems machine feature for that.

Today I just gave Colima a shot with Incus and I am very impressed! In particular as it seems you have brought ZFS storage with Colima which I could not get with Orbstack becasue they don’t allow me to build a kernel module.

Please continue this great work :)!

2 Likes
$ head /etc/lsb-release /etc/os-release /etc/debian_version
head: cannot open '/etc/lsb-release' for reading: No such file or directory
==> /etc/os-release <==
PRETTY_NAME="Debian GNU/Linux trixie/sid"
NAME="Debian GNU/Linux"
VERSION_CODENAME=trixie
ID=debian
HOME_URL="https://www.debian.org/"
SUPPORT_URL="https://www.debian.org/support"
BUG_REPORT_URL="https://bugs.debian.org/"

==> /etc/debian_version <==
trixie/sid

Hmm, like you say then, weird that it’s a Debian image with an Ubuntu kernel. Maybe they did that to get ZFS, but then why not Ubuntu top-to-bottom?

1 Like

When I run containers on Linux, they usually point to the kernel on the host. In this case, instead of pointing to Darwin the uname command in the container points to Linux Trixie 6.8.0-39-generic #39-Ubuntu SMP PREEMPT_DYNAMIC which obviously is a Linux kernel (apparently with an “Ubuntu” tag from Debian.) Where is that coming from? Is Incus providing this?

When you launch Incus containers and VMs, those run inside the LinuxVM. When you launch an Incus VM, then this VM runs inside the Linux VM (nested virtualization).

To me this seems logical. I perhaps was confused by the discussion in this thread on the patch bringing in “nested virtualization”. If I’m running a Linux VM, perhaps via QEMU since that is installed, then don’t we already have nested virtualization?

There are no surprises there as the container (debian) is using the host’s kernel (ubuntu).

This has been previously answered. Incus does not run directly on macOS, but rather via a Linux virtual machine.

1 Like