How to pass a file to a VM in IncusOS? (For passing init files like the Ignition config e.g. for CoreOS)

IncusOS
,
1

use case

I tried installing Fedora CoreOS as a VM to then run a container inside it with podman on IncusOS.

resources/related posts/research

I tried looking at these posts:

The last thing is most interesting:

This is about the mechanism we would use to pass the Ignition config to the instance. In this case we would likely use the same option used to pass cloud-init configs: How to use cloud-init - Incus documentation

The link looks interesting, so one would need to use cloud-init

However:

Maybe the OpenStack image would work as is as it has support for cloud-init drives from memory.

So it seems only(?) this image includes cloud-init for it to work, actually? I used the qemu ones (see below), so using different ones just for cloud-init kinda feels like “misusing” the images. :see_no_evil_monkey:

what I tried so far

You can see my progress in this repo:

Basically, currently I just use the qemu images, package them as a tar.xz and push them to the CoreOS host. And this works flawlessly.
This follows this awesome blog post by ignisc that already explains a lot:

problems

I do not (only) run Incus, but IncusOS. That means, I have no access to the file system and cannot run terminal commands on that host.

The blog post linked above, however, then tries to pass the Ignition config as a qemu variable:

incus config set fcos02 raw.qemu=" -fw_cfg name=opt/com.coreos/config,file=/var/lib/incus/ignition/fcos02.ign"

Obviously, this does not work on a remote IncusOS instance/server. You also cannot place the file “on the server” to adjust the path.

Questions

  • Is there any way to pass an init file like the mentioned one to the server?

  • What do you think of using the OpenStack image, as indicated? (And then cloud-init? Somehow?)

    • So if cloud-init is used, how to do that actually? I read the docs, but I found no way of placing a file somewhere (the Ignition file) to later be picked up by cloud-init.

  • The blog post says, at the end:

    Find a way not to have to scp the ignition files (maybe using a volume containing them all ?)

    This confuses me a bit, because I do not actually see scp mentioned in the guide before at all and how it should work.

Cross-posted to the Fedora CoreOS community forum.

2

Oh I now found the incus file push command, but I always get an error it would not be running – while actually it is:

$ incus file push ./server-pihole.ign.json IncusOS:fcos-pihole/var/lib/incus/ignition/fcos.ign
Error: Failed getting instance SFTP connection: VM agent isn't currently running
$ incus info IncusOS:fcos-pihole 
Name: fcos-pihole
Description: 
Status: RUNNING
Type: virtual-machine
[…]

Edit: Ah I just read that (somewhat hidden) sentence in the docs, that explains it and obviously makes it more complicated:

For virtual machines, the incus-agent process must be running inside of the virtual machine for them to work.

So we have a chicken and egg problem here. I could install the agent with a (first butane) and then ignition config, but I cannot do this, because I cannot push the config file to the server. :sweat_smile:

3

We support cloud-init which can then be exposed as a cdrom device by doing incus config device add VM cloud-init disk source=cloud-init:config and we also support the systemd credentials mechanism which may also be able to provide initial provisioning data to your VM.

Once you have an incus-agent up and running in there, then you have a bunch more options from using incus file pull/push to attaching a filesystem custom volume and then reading files from that.

4

Just tinkered with this all day so thought I’d share the fruits of my tokens.

The openstack image wants to see an attached volume specifically named config-2 with the ignition located at /configdrive/openstack/latest/user_data, not the normal cloud-init/NoCloud structure. So I created a .iso as it wished and mounted that, which ended up working.

Still trying to get it to recognize the incus-agent.