RHEL VMs with LXD, no datasource


I am trying to figure out a way to make cloud-init work with RHEL VMs under LXD.
I tried different LXD releases from 4.0 to 5.2 (snap based on Ubuntu 20.04 OS) and none of them made cloud-init work inside my RHEL8 VM.
Taking a look at the VM itself, I see that cloud-init starts (cloud-init-21.1-15.el8.noarch on an officiel RHEL 8.6 cloud image from RedHat portal) but it detects no datasource.
/dev/lxd/sock is not there but /sys/class/dmi/id/board_name matches “LXD”.
“security.devlxd” is set at the instance configuration level.

What can be done ?
Is there something to tweak in the RHEL image or a misconfiguration ?

I tried with RHEL9 images too but no more luck :confused:

Anything you’ve come across in your cloud-init work @monstermunchkin ?

Have you tried using the special cloud-init disk source?

lxc config device add <instance> disk source=cloud-init:config

This should give you the NoCloud data source.

@monstermunchkin this is where I ended.

I’m deploying these VMs with Terraform and the LXD provider but the documentation is very slight (to say the least) and I didn’t know, at first, how to add a config drive.
Now that I managed to do it, it works way better and I will stick to this method because it is a lot more common that native LXD datasource.
Anyway, just FYI, I discovered why /dev/lxd/sock wasn’t inside the VM, it was because SELinux was configured in “enforced” mode by default in the official RHEL cloud image.
I switched it to “permissive” and then , I could see the socket and even mount the virtiofs share for the agent.
However, it seems that the agent is not automatically installed by cloud-init, I had to add a few run-command in the user-data to force the install and then it worked but we’re fare from an “out of the box” experience :confused:

The LXD datasource is fairly new, and only works if /dev/lxd/sock is present, and the lxd-agent is running.

Is it not mounted automatically?

cloud-init isn’t responsible for installing the lxd-agent. That is done by LXD itself using the config drive.

“Is it not mounted automatically?”

No it wasn’t, that’s why I added this in my user-data :


  • mount -t virtiofs config /mnt; sleep5; cd /mnt; ./install.sh; cd /; umount /mnt
  • systemctl start lxd-agent

Thats expected, recent official Ubuntu images (e.g. lxc launch ubuntu:22.04) and the LXD project’s images (e.g. lxc launch images:ubuntu/22.04) come with inbuilt systemd units that attempt to mount and start the lxd-agent is available.

But these would not be in custom built images/VMs.

1 Like

That’s unfortunate, but you could always try our AlmaLinux or Rocky Linux images.

lxc launch images:almalinux/9/cloud v1 --vm or lxc launch images:rockylinux/8/cloud v1 --vm.

Agreed but unfortunately I can’t … you can imagine that RHEL is an enterprise choice, not specifically mine.
However, after a bit of struggle, I managed to build a customized RHEL7 image with UEFI support enabled and now it works.
Thanks anyway

Unfortunately we cannot distribute modified RHEL images due to licensing restrictions.

Yup, I know …

Anyway, now that I managed to build RHEL images, I have one last issue :
With RHEL8+, virtiofs is enabled in the kernel and even if it needs some tricks to make it work as expected, I can install the lxd-agent and at the end, LXD is happy with the IP address reported to LXC.

However, RedHat was a bit annoying with RHEL7 because it disabled 9p and virtiofs in their kernel and there is no way to mount the virtiofs/9p shre inside the VM and then, install the LXD agent.
Because of that, the VM works as expected but lxc ls doesn’t display the IP address of the VM.
This is not something that we can’t live with but it makes “wait_for_network” option in terraform unusable and terraform ends in error because it thinks the VM is not “alive”.

Of course, we could just disable the “wait_for_network” option but I find this IP report quite usefull to make sure a VM is properly up and I would think it would be too bad to discard it.
Is there a way to install the lxd-agent manually in anyway ?
Are there other possibilities to report this information to LXD ?
How does it work with *BSD or Windows based systems ?
Can’t we rely on the qemu-guest-agent instead which is a way more common package that most of the systems (even Windows) has ?

Thanks for your help

Does RHEL7 not have virtiofs? I know CentOS disabled 9p, but we found that virtiofs works OK, which is why our RHEL7 derived VM images work with the lxd-agent (as its exported over both protocol types).

Is there any in the system logs about 9p or virtiofs from LXD?

No, because they are totally different pieces of software that use different underlying protocols.

Absolutely yes, you can copy it into the image manually from the host system.

virtiofs is indeed disabled
I tried both mounts (virtiofs and 9p), none of them work.

It seems CentOS team took some freedom when they repacked the RHEL packages (and I would say, I would love to see RedHat taking the same path sometimes …)

What are the requirements to make it work ?
Does it need /dev/lxd/sock ?

It needs the vhost_vsock kernel module available in both host and guest.
Other than that it is a static binary.

I’ll do some testing and keep you posted

1 Like