I am trying to figure out a way to make cloud-init work with RHEL VMs under LXD.
I tried different LXD releases from 4.0 to 5.2 (snap based on Ubuntu 20.04 OS) and none of them made cloud-init work inside my RHEL8 VM.
Taking a look at the VM itself, I see that cloud-init starts (cloud-init-21.1-15.el8.noarch on an officiel RHEL 8.6 cloud image from RedHat portal) but it detects no datasource.
/dev/lxd/sock is not there but /sys/class/dmi/id/board_name matches “LXD”.
“security.devlxd” is set at the instance configuration level.
What can be done ?
Is there something to tweak in the RHEL image or a misconfiguration ?
I’m deploying these VMs with Terraform and the LXD provider but the documentation is very slight (to say the least) and I didn’t know, at first, how to add a config drive.
Now that I managed to do it, it works way better and I will stick to this method because it is a lot more common that native LXD datasource.
Anyway, just FYI, I discovered why /dev/lxd/sock wasn’t inside the VM, it was because SELinux was configured in “enforced” mode by default in the official RHEL cloud image.
I switched it to “permissive” and then , I could see the socket and even mount the virtiofs share for the agent.
However, it seems that the agent is not automatically installed by cloud-init, I had to add a few run-command in the user-data to force the install and then it worked but we’re fare from an “out of the box” experience
Thats expected, recent official Ubuntu images (e.g. lxc launch ubuntu:22.04) and the LXD project’s images (e.g. lxc launch images:ubuntu/22.04) come with inbuilt systemd units that attempt to mount and start the lxd-agent is available.
But these would not be in custom built images/VMs.
Agreed but unfortunately I can’t … you can imagine that RHEL is an enterprise choice, not specifically mine.
However, after a bit of struggle, I managed to build a customized RHEL7 image with UEFI support enabled and now it works.
Thanks anyway
Anyway, now that I managed to build RHEL images, I have one last issue :
With RHEL8+, virtiofs is enabled in the kernel and even if it needs some tricks to make it work as expected, I can install the lxd-agent and at the end, LXD is happy with the IP address reported to LXC.
However, RedHat was a bit annoying with RHEL7 because it disabled 9p and virtiofs in their kernel and there is no way to mount the virtiofs/9p shre inside the VM and then, install the LXD agent.
Because of that, the VM works as expected butlxc ls doesn’t display the IP address of the VM.
This is not something that we can’t live with but it makes “wait_for_network” option in terraform unusable and terraform ends in error because it thinks the VM is not “alive”.
Of course, we could just disable the “wait_for_network” option but I find this IP report quite usefull to make sure a VM is properly up and I would think it would be too bad to discard it.
Is there a way to install the lxd-agent manually in anyway ?
Are there other possibilities to report this information to LXD ?
How does it work with *BSD or Windows based systems ?
Can’t we rely on the qemu-guest-agent instead which is a way more common package that most of the systems (even Windows) has ?
Does RHEL7 not have virtiofs? I know CentOS disabled 9p, but we found that virtiofs works OK, which is why our RHEL7 derived VM images work with the lxd-agent (as its exported over both protocol types).
Is there any in the system logs about 9p or virtiofs from LXD?
virtiofs is indeed disabled
I tried both mounts (virtiofs and 9p), none of them work.
It seems CentOS team took some freedom when they repacked the RHEL packages (and I would say, I would love to see RedHat taking the same path sometimes …)