OpenWRT testing in Incus

[owrtinincus (Nathan Tweed)]

I plan to start testing and experimenting with OpenWRT builds in Incus and am curious if anyone has tried the same?

Of special interest to me is whether anyone has successfully complied Incus Agent to run on OpenWRT, be it x86-64 or ARM64 (AFAIK, there’s no Incus Agent for OpenWRT, even for x86-64?).

Related:

Incus-agent in OpenWRT VM

I need prepare managed OpenWRT VM running in the Incus deployment. I have built custom OpenWRT image, converted it and import to the Incus as VM, but I need some help to debug communication between Incus daemon and and incus-agent in the VM. What is done: Custom OpenWRT image with vsock.ko kernel module and custom network setup built and imported as incus VM. VM run, incus-agent virtual disk mounted into the VM (mount -t 9p config /mnt) incus-agent started with debug and verbose options (inc…

Which 9p/other kernel modules are required for incus-agent?

I saw some threads on setting up OpenWRT for it. I was curious and looked here: https://openwrt.org/packages/table/start?dataflt[Name_pkg-dependencies*~]=9p kmod-9pnet and kmod-fs-9p are available, so I should be able to mount the agent disk without needing a custom volume, no? What other support do I need in the kernel for it to work? Over what qemu-guest-agent needs, that’s available so its needs are covered.

Upgrading OpenWrt

Does anyone who use the OpenWrt container have a script they can share for upgrading OpenWrt that creates a new container, copies over the config/user packages that works well with Incus?

Incus vm image request: OpenWRT (for x86_64 amd64) on images.linuxcontainers.org

Hi, I’ve been using the openwrt image for lxc containers for a while and it’s been great. I’m running now into an issue where I need to tune the kernel for certain openwrt features and I can’t tune the host’s kernel for that. So, I’m attempting to use an incus vm for it, however, the openwrt image is only available for containers, not vms. Would it be possible to add a build for openwrt vm images?

[tregubovav (@AT)]

1. OpenWRT container has restricted functionality. You are not able to load custom OpenWRT kernel modules (kmods).
2. Default OpenWRT images can be run as Incus/LXD VM on X86 and arm64 platforms. However, the Incus Agent can’t communicate with the incus daemon running on host. This restricts using incus to orchestrate OpenWRT VM instances.
3. There is a discussion at OpenWRT Forum: Vsock.ko kernel module needed (aarch64/X86_64 targets) and Pull Request created that allows to run OpenWRT VM under the incus natively. However, there is almost no progress due to low interest to this topic.

[owrtinincus (Nathan Tweed)]

@tregubovav

OpenWRT container has restricted functionality. You are not able to load custom OpenWRT kernel modules (kmods)

Thank You - apologies if I’m being pedantic in your use of the word OpenWRT container but does the inablility to load custom OpenWRT kernel modules (kmods) also apply to OpenWRT VMs?

@tregubovav

However, the Incus Agent can’t communicate with the incus daemon running on host. This restricts using incus to orchestrate OpenWRT VM instances

Noted. I have been able to work around file push/pull using SSH and SCP. I’m working on getting shared p9 working (I will have a thread up soon) - I have it working using qemu-system-x86_64 like below, need to get it working under incus next:

qemu-system-x86_64 \
  -enable-kvm -cpu host -m 2048 -L "$BIOS_DIR" \
  -kernel "$KERNEL_PATH" \
  -append "$APPEND_ARGS" \
  -drive file="$DISK_PATH",format=raw,if=virtio \
  -virtfs local,path="$HOST_SCRIPT_DIR",mount_tag=host_share,security_model=none,id=host_share \
  -chardev socket,id=char0,path="$SERIAL_SOCK",server=on,wait=off,logfile="$DEBUG_LOG" \
  -serial chardev:char0 \
  -display none

where APPEND_ARGS has a script that starts with mount -t 9p -o trans=virtio host_share /mnt

@tregubovav

There is a discussion at OpenWRT Forum: Vsock.ko kernel module needed (aarch64/X86_64 targets) and Pull Request created that allows to run OpenWRT VM under the incus natively. However, there is almost no progress due to low interest to this topic.

Thank You for the link and the PR kernel: modules to run OpenWRT VM in LXD or incus by tregubovav-dev · Pull Request #19654 · openwrt/openwrt · GitHub

[tregubovav (@AT)]

@owrtinincus

Thank You - apologies if I’m being pedantic in your use of the word OpenWRT container but does the inablility to load custom OpenWRT kernel modules (kmods) also apply to OpenWRT VMs?

This is container’s nature. In simple word container run the process(es) in restricted linux environment unless they run as privilege container (i.e. run under root on the host instance). Even if you run the OpenWRT container as privilege container, you will not able to load OpenWRT kernel modules because they are not compatible with your host OS.

@owrtinincus

Noted. I have been able to work around file push/pull using SSH and SCP…

My experience shows that direct conversion the default OpenWRT X86_64 or armv8 allows incus to create and start instances from it. However, you need to have a custom OpenWRT network in the image or have ability to change it directly at the incus instance’s root storage (depending on kind of incus storage you use) . You need also configure incus network(s) for the instances. After that you can get network access to the OpenWRT instance. However, the incus will not able to manage it and you could see some side effects like long shutdowns. You will have to fine tune incus instance’s properties to reduce collateral damages with running unmanaged VM under incus.

[owrtinincus (Nathan Tweed)]

@tregubovav

you will not able to load OpenWRT kernel modules because they are not compatible with your host OS.

Agreed. I was just clarifying this won’t be an issue in VMs because the kernel can be different from the host

@tregubovav

have ability to change it directly at the incus instance’s root storage (depending on kind of incus storage you use)

Yes, I created a separate thread for it to keep this one more focused: Root volume protection in Incus VM

[argon5 (Oleksandr)]

I had tried to run OpenWRT as an Incus container about a year ago as my home lab’s router, and after a day or two of tinkering with it, I hit quite hard a dead end. I basically learned that OpenWRT in containers is a unviable setup and an extremely bad idea for many reasons.

Containers are ideal for user-land applications that are kernel agnostic and use a common kernel API, while a kernel itself belongs to a host and is shared among all containers, so containerized apps shall not do its heavy tweaking if any at all; otherwise, containerization is pointless.

OpenWRT is a router system where management apps configure kernel modules and network subsystems, expecting that the whole kernel is their exclusive responsibility. This violates a fundamental concept of containers “being a kernel agnostic” as it requires the host’s kernel to be modified beyond acceptable limits. As a result, I had a dozen issues related to missing modules in the kernel, like VPNs, PPP, tunnels, etc. All of them you must bring to the host kernel, and some of them even passthrough into a container, which is bad setup in many ways. Furthermore, OpenWRT does patching some modules for its own kernel, so using a host’s kernel with different modules may result in unpredictable behavior and bugs. Finally, I was unable to set up NTP client/server daemons as they require access to manage kernel’s time, and the kernel doesn’t allow this for containers out of security reasons.

At this moment Incus does not have OpenWRT VM images, and related activities are more than year in progress. Given that, I don’t see why OpenWRT as Incus container image should exist at all and confuse people, as it’s barely usable for real-world scenarios. It must be only in a form of a VM with its own isolated kernel.

[owrtinincus (Nathan Tweed)]

I didn’t face any of the issues you mention but I also didn’t attempt to run it as a container. The beauty about Incus is it supports both containers as well as VMs with a nice abstraction.

Being able to run custom OpenWRT VMs using Incus has improved my life substantially because I can now run integration tests for my DIY home routers before I even have to actually flash a real router.

[argon5 (Oleksandr)]

@owrtinincus

I didn’t face any of the issues you mention but I also didn’t attempt to run it as a container.

That was the whole point of my post: run OpenWRT as VM only, not container, so that’s why you didn’t face issues. You ran it in a proper way. My second point was that Incus shall drop OpenWRT container images it currently offers as being useless/confusing and start officially support VM images only which it doesn’t support now.

[argon5 (Oleksandr)]

@tregubovav

There is a discussion at OpenWRT Forum: Vsock.ko kernel module needed (aarch64/X86_64 targets) and Pull Request created that allows to run OpenWRT VM under the incus natively. However, there is almost no progress due to low interest to this topic.

BTW, this mentioned PR was closed because a similar PR adding vsock was merged in. Does this open a path for Incus agent in OpenWRT VM image?

[owrtinincus (Nathan Tweed)]

@argon5

Does this open a path for Incus agent in OpenWRT VM image?

Yes it does - because I already have a SSH/SCP and volume mount path setup (to work around the previous lack of agent) I haven’t tried it out myself but the other PR that was merged in implies that new builds and some modules should allow incus agent to work just fine. I might be missing some details like whether incus agent exists for ARM (and that might be irrelevant if you’re using only AMD64)

If you would like to report back findings, that would be really awesome as this brings OpenWRT VMs (atleast AMD64) inline with most modern OSes in incus