OpenWRT testing in Incus

[owrtinincus (Nathan Tweed)]

I plan to start testing and experimenting with OpenWRT builds in Incus and am curious if anyone has tried the same?

Of special interest to me is whether anyone has successfully complied Incus Agent to run on OpenWRT, be it x86-64 or ARM64 (AFAIK, there’s no Incus Agent for OpenWRT, even for x86-64?).

Related:

Incus-agent in OpenWRT VM

I need prepare managed OpenWRT VM running in the Incus deployment. I have built custom OpenWRT image, converted it and import to the Incus as VM, but I need some help to debug communication between Incus daemon and and incus-agent in the VM. What is done: Custom OpenWRT image with vsock.ko kernel module and custom network setup built and imported as incus VM. VM run, incus-agent virtual disk mounted into the VM (mount -t 9p config /mnt) incus-agent started with debug and verbose options (inc…

Which 9p/other kernel modules are required for incus-agent?

I saw some threads on setting up OpenWRT for it. I was curious and looked here: https://openwrt.org/packages/table/start?dataflt[Name_pkg-dependencies*~]=9p kmod-9pnet and kmod-fs-9p are available, so I should be able to mount the agent disk without needing a custom volume, no? What other support do I need in the kernel for it to work? Over what qemu-guest-agent needs, that’s available so its needs are covered.

Upgrading OpenWrt

Does anyone who use the OpenWrt container have a script they can share for upgrading OpenWrt that creates a new container, copies over the config/user packages that works well with Incus?

Incus vm image request: OpenWRT (for x86_64 amd64) on images.linuxcontainers.org

Hi, I’ve been using the openwrt image for lxc containers for a while and it’s been great. I’m running now into an issue where I need to tune the kernel for certain openwrt features and I can’t tune the host’s kernel for that. So, I’m attempting to use an incus vm for it, however, the openwrt image is only available for containers, not vms. Would it be possible to add a build for openwrt vm images?

[tregubovav (@AT)]

1. OpenWRT container has restricted functionality. You are not able to load custom OpenWRT kernel modules (kmods).
2. Default OpenWRT images can be run as Incus/LXD VM on X86 and arm64 platforms. However, the Incus Agent can’t communicate with the incus daemon running on host. This restricts using incus to orchestrate OpenWRT VM instances.
3. There is a discussion at OpenWRT Forum: Vsock.ko kernel module needed (aarch64/X86_64 targets) and Pull Request created that allows to run OpenWRT VM under the incus natively. However, there is almost no progress due to low interest to this topic.

[owrtinincus (Nathan Tweed)]

@tregubovav

OpenWRT container has restricted functionality. You are not able to load custom OpenWRT kernel modules (kmods)

Thank You - apologies if I’m being pedantic in your use of the word OpenWRT container but does the inablility to load custom OpenWRT kernel modules (kmods) also apply to OpenWRT VMs?

@tregubovav

However, the Incus Agent can’t communicate with the incus daemon running on host. This restricts using incus to orchestrate OpenWRT VM instances

Noted. I have been able to work around file push/pull using SSH and SCP. I’m working on getting shared p9 working (I will have a thread up soon) - I have it working using qemu-system-x86_64 like below, need to get it working under incus next:

qemu-system-x86_64 \
  -enable-kvm -cpu host -m 2048 -L "$BIOS_DIR" \
  -kernel "$KERNEL_PATH" \
  -append "$APPEND_ARGS" \
  -drive file="$DISK_PATH",format=raw,if=virtio \
  -virtfs local,path="$HOST_SCRIPT_DIR",mount_tag=host_share,security_model=none,id=host_share \
  -chardev socket,id=char0,path="$SERIAL_SOCK",server=on,wait=off,logfile="$DEBUG_LOG" \
  -serial chardev:char0 \
  -display none

where APPEND_ARGS has a script that starts with mount -t 9p -o trans=virtio host_share /mnt

@tregubovav

There is a discussion at OpenWRT Forum: Vsock.ko kernel module needed (aarch64/X86_64 targets) and Pull Request created that allows to run OpenWRT VM under the incus natively. However, there is almost no progress due to low interest to this topic.

Thank You for the link and the PR kernel: modules to run OpenWRT VM in LXD or incus by tregubovav-dev · Pull Request #19654 · openwrt/openwrt · GitHub

[tregubovav (@AT)]

@owrtinincus

Thank You - apologies if I’m being pedantic in your use of the word OpenWRT container but does the inablility to load custom OpenWRT kernel modules (kmods) also apply to OpenWRT VMs?

This is container’s nature. In simple word container run the process(es) in restricted linux environment unless they run as privilege container (i.e. run under root on the host instance). Even if you run the OpenWRT container as privilege container, you will not able to load OpenWRT kernel modules because they are not compatible with your host OS.

@owrtinincus

Noted. I have been able to work around file push/pull using SSH and SCP…

My experience shows that direct conversion the default OpenWRT X86_64 or armv8 allows incus to create and start instances from it. However, you need to have a custom OpenWRT network in the image or have ability to change it directly at the incus instance’s root storage (depending on kind of incus storage you use) . You need also configure incus network(s) for the instances. After that you can get network access to the OpenWRT instance. However, the incus will not able to manage it and you could see some side effects like long shutdowns. You will have to fine tune incus instance’s properties to reduce collateral damages with running unmanaged VM under incus.

[owrtinincus (Nathan Tweed)]

@tregubovav

you will not able to load OpenWRT kernel modules because they are not compatible with your host OS.

Agreed. I was just clarifying this won’t be an issue in VMs because the kernel can be different from the host

@tregubovav

have ability to change it directly at the incus instance’s root storage (depending on kind of incus storage you use)

Yes, I created a separate thread for it to keep this one more focused: Root volume protection in Incus VM