How to run Fedora CoreOS as an LXD VM

Tutorials
1

Hello,

Recently, I played a bit with Fedora CoreOS on my homelab LXD server and I thought it would be interesting to share my experience with it.

Here the configuration I used :

  • LXD host : Ubuntu 22.04 LTS
  • LXD version : 5.2
  • LXD storage driver : ZFS

Image preparation

As you may noticed, there is no images available for Fedora CoreOS on the LXD images servers, but this isn’t a problem to run this system on LXD.

  • Firstly, you need to download a QCOW2 image of Fedora CoreOS on the official website : Download Fedora CoreOS.

    You need to download the QEMU version.

  • Once the image is downloaded, you need to extract the xz tarball to get the QCOW2 image :

    $ unxz fedora-coreos-36.20220522.3.0-qemu.x86_64.qcow2.xz

  • In order to import a custom image in the LXD image store, you need to pass a YAML metadata file when importing the QCOW2 image. Here is an example :

    architecture: x86_64
creation_date: 1655660820 # this is an epoch timestamp
properties:
  description: Fedora CoreOS
  os: fedora-coreos
  release: 36

  • Moreover, this file must be passed as a tarball in the lxc import command :

    $ tar cf metadata.tar metadata.yaml

  • Finally, you can import the image in your local LXD image store :

    $ lxc image import metadata.tar fedora-coreos-36.20220522.3.0.qcow2 --alias fcos/36

Prepare the Ignition file

As you may know, Fedora CoreOS requires an Ignition file to configure the instance on the first boot. If you don’t configure the instance with an Ignition file, you may not be able to connect to the instance. This is more or less similar to Cloud-Init, but this tool is mandatory for CoreOS-like operating system such as Fedora CoreOS or Flatcar Linux.

  • Before creating an Ignition file for our instance, you need a tool called Butane. Butane is a tool to convert Ignition file from an human-readable format (YAML) into a machine-readable format file (JSON). You can download the binary here : Releases · coreos/butane · GitHub

  • Once you have download and installed Butane, you can create your first Ignition file in YAML format. This example is basic, we will only set our public SSH key for the core user (the Fedora CoreOS default user) and give it a dedicated hostname :

    ---
variant: fcos
version: 1.4.0
passwd:
  users:
  - name: core
    ssh_authorized_keys:
    - <your public ssh key>
storage:
  files:
  - path: /etc/hostname
    mode: 0644
    contents:
      inline: fcos-lxd

    The complete reference for Ignition : Configuration specifications - coreos/butane.

    You can also find some examples here : https://docs.fedoraproject.org/en-US/fedora-coreos/.

  • Then, you need to convert this YAML file into a JSON file using Butane :

    $ butane --pretty --strict fcos-lxd.yaml > fcos-lxd.ign

  • The file you generated should looks like this :

    {
  "ignition": {
    "version": "3.3.0"
  },
  "passwd": {
    "users": [
      {
        "name": "core",
        "sshAuthorizedKeys": [
          " <your public ssh key>"
        ]
      }
    ]
  },
  "storage": {
    "files": [
      {
        "path": "/etc/hostname",
        "contents": {
          "source": "data:,fcos-lxd"
        },
        "mode": 420
      }
    ]
  }
}

It’s ready !

Launch the instance

Once your Fedora CoreOS image is imported and your Ignition file is ready, you can launch the instance. However, the instance launch requires few configurations.

In order for the Ignition file to be read by your instance, you need to pass it into the QEMU firmware configuration device of your instance. LXD does not expose this feature to the user, so we need to use the raw.qemu option in the command-line :

$ lxc launch fcos/36 fedora-coreos --vm \
-c security.secureboot=false \
-c raw.qemu="-fw_cfg name=opt/com.coreos/config,file=/var/lib/snapd/hostfs/<path/to/fcos-lxd.ign>" \
-c raw.apparmor="/var/lib/snapd/hostfs/<path/to/fcos-lxd.ign> r,"

As you may notice, we need to specify a specific path to our Ignition file with a dedicated AppArmor rule. The QEMU binary used is confined inside the LXD snap mount namespace, and therefore can’t access to the host filesystem natively.

However, there is a trick, you can access to your Ignition file using a special path exposed inside the snap mount namespace using the /var/lib/snapd/hostfs prefix. This is some kind of direct access to the host filesystem, but the access will be denied by AppArmor by default, that’s why you need to add a custom AppArmor rule to the lxc launch command-line :

-c raw.apparmor="/var/lib/snapd/hostfs/<path/to/fcos-lxd.ign> r,"

This rule defines that the QEMU process of our instance is able to read (r for read) our Ignition file on the host filesystem. The comma at the end of the rule is important, don’t forget it !

Finally, you can launch the instance and check if everything is alright :

$ ssh core@10.0.122.120
Fedora CoreOS 36.20220522.3.0
Tracker: https://github.com/coreos/fedora-coreos-tracker
Discuss: https://discussion.fedoraproject.org/tag/coreos

[core@fcos-lxd ~]$

LXD Agent

By default, no LXD agent is available in Fedora CoreOS. The installation script provided through 9p/VirtioFS mount won’t work as the latter requires to write some files in filesystem directories that are read-only (/usr especially).

Therefore, this section is not available for now, I still need to figure how to configure the agent inside the instance, but also how to configure SELinux since it doesn’t like the LXD agent :frowning:

That’s it, hope this tutorial will be useful :wink:

1 Like