Scriptlet to override the QEMU boot configuration

Incus
1

It seems like I will need to write a scriptlet so I can override the kernel arguments to have incus achieve something analogous to:

APPEND_ARGS="root=PARTUUID=13d8cd4e-02 rw rootwait console=ttyS0,115200 earlyprintk=ttyS0,115200 init=/bin/sh /mnt/bootstrap.sh"

qemu-system-x86_64 \
  -enable-kvm -cpu host -m 2048 -L "$BIOS_DIR" \
  -kernel "$KERNEL_PATH" \
  -append "$APPEND_ARGS" \
  -drive file="$DISK_PATH",format=raw,if=virtio \
  -virtfs local,path="$HOST_SCRIPT_DIR",mount_tag=host_share,security_model=none,id=host_share \
  -chardev socket,id=char0,path="$SERIAL_SOCK",server=on,wait=off,logfile="$DEBUG_LOG" \
  -serial chardev:char0 \
  -display none

Trying to hack via setting raw.qemu has been a dead end for me.

I don’t care about exactly replicating the above qemu-system-x86_64: really looking for the ability to set up a socket that can be accessed from the host and run bootstrap.sh instead of the usual init. host_share would be a plus!

The VM is for a custom openwrt x86-64 build, so $KERNEL_PATH and $DISK_PATH does require a legacy BIOS boot and are incompatible with UEFI

I’ve been searching for scriptlet examples and wasn’t able to arrive at anything that remotely works without looking horrendous and unmaintainable - are there samples for me to iterate from using Incus 6.20?

The latest “clean” iteration I arrived at before I ended up with something I’m ashamed to put online (error was Failed running QEMU scriptlet at config stage: Failed to run: Addition or deletion of -bios or -kernel is unsupported from incus/internal/server/scriptlet/qemu.go at 112678ceca4673d01d23332ab666138e1adad1c4 · lxc/incus · GitHub ):

def qemu_hook(instance, stage):
    # seems like only the 'config' stage allows us to modify the command line
    if stage == "config":
        log_info("Scriptlet: Accessing QEMU command line at config stage")

        # Capture the original command line
        args = get_qemu_cmdline()
        log_info("BEFORE: " + str(args))

        # Inject our legacy boot parameters
        args.extend(["-kernel", "$KERNEL_PATH"])
        args.extend(["-append", "$APPEND_ARGS"])
        # etc

        # Log the modified state
        log_info("AFTER:  " + str(args))

        # Commit the changes
        set_qemu_cmdline(args)
        
        log_info("set_qemu_cmdline applied successfully in config stage")

If on the other hand, this is expected behavior, what’s the workflow if I wanted to experiment with kernels and tuning their parameters using Incus 6.20 VM?

Now I’m using a OpenWRT VM to achieve all of this so if this detail matters or is relevant, let me know as well, but so far it feels like the protections are from Incus

2

We’ve been meaning to work on that for quite some time, but yeah, the docs are incomplete.

This check comes from this discussion (if GitHub doesn’t scroll you to the comment, search -kernel).

Yep, it is expected behavior; you should set -kernel and -bios arguments in raw.qemu and that should be fine. Be careful though, as you should not be able to use environment variables there. If you really need variability, you can read user.* config values within your scriptlet (but not within raw.qemu).

Hope that helps.

1 Like
3

Thank You - it’s actually fantastic that I get to discuss direct with the author because then we can engage in the why and what. For anyone else following this discussion, at that comment you wrote:

Also, I propose to prevent messing with -bios or -kernel through set_qemu_cmdline, because generateQemuConfigFile checks these and I really don’t want to change the logic of the function (just rewire it a bit so I can plug the scriptlet before WriteFile). So basically, we shall check the presence of -bios and -kernel before and after, and both should match. generateQemuConfigFile only checks their presence to load the NVRAM and setup boot priorities, so we can mess with the values as long as it’s already there.

I’m wondering if it’s acceptable for someone experimenting or developing kernels and OS to use Incus as a declarative layer (vs. using a hypervisor like qemu-system-x86_64 directly)?

If so, I imagine these people will need to tweak -bios and -kernel, right?

Now if the suggestion you’re making is, to make those changes directly in raw.qemu, the reason why I had to go the scriptlet route in the first place was incus is doing something behind the scenes that effectively either shadow or throw away what I set in -kernel in raw.qemu

If this is unexpected, I think I will open a new thread with ready to go bash scripts, but wanted to clarify whether it’s known that incus could either shadow or throw away settings in -kernel and -append in raw.qemu set like so:

incus config set "$VM_NAME" raw.qemu -- "-kernel foo -append bar"

For example: If I literally set this command, I would expect VM_NAME to never find its kernel (unless it was literally foo)

4

Well it certainly depends on the features you want. I like to tinker with the lower levels, and some of my contributions to Incus are actually here to support that. So for me, it works, and when it doesn’t, I contribute :slight_smile:

As I never develop for non-UEFI targets, I never have to tweak these options. And I think you should go the UEFI route as much as possible.

But why would you write overrides in raw.qemu in the first place if you’re going to shadow them later?

Doing it should append -kernel foo -append bar to the QEMU command line.

Are you getting different results?

1 Like
5

It looks like I confused you. If I can get the overrides in raw.qemu to work, I don’t need the the scriptlet. I had to go the scriptlet route in the first place because it seems like incus is doing something behind the scenes that effectively either shadow or throw away what I set in -kernel and -append in raw.qemu

Yes, when I set incus config set "$VM_NAME" raw.qemu -- "-kernel foo -append bar", my VM still continues to boot the original kernel. I will open a new thread with ready to go bash scripts, because that will have nothing to do with scriptlets.

PS: I was able to write a scriptlet that managed to override -kernel but won’t share it here because it’s clear incus doesn’t want a scriptlet that to override -kernel. however, this allowed me to boot my system exactly like I wanted to while the raw raw.qemu fails as explained above.

6

Oh, right!
Well, raw.qemu appends to the command-line arguments (see here) and I don’t think Incus messes with them any further except for clock adjustments. I’m afraid I can’t help you debug that any further as I’m not really knowledgeable in QEMU’s Direct Linux Boot feature. But what you set really should be what you get…

Maybe additional logs (incus monitor) could help pin down the issue.

Overriding it is really fine, but for technical reasons, you can’t add (if not previously defined) or delete (if previously defined) -kernel and -boot arguments in a scriptlet, only change them. The reason is that a rather complex machinery is there to properly configure the VM depending on the presence of these arguments, and this machinery is run before the scriptlet logic is triggered.

So, please share your scriptlet! You never know how many people that could help in the future.

1 Like
7

Thank You. I will use that in my new post (in progress)

Yes, that’s precisely the trick I used. Placed literally the command (incus config set "$VM_NAME" raw.qemu -- "-kernel foo) and updated -kernel from the scriptlet.

Now since raw.qemu is expected to work literally, I’m doing something wrong, writing an unnecessary scriptlet, so down to it!

Again, Thank You