/proc/kallsyms missing in some lxd virtual machines

LXD
1

Hi,

I need to do some performance debugging using perf on lxd virtual machines, and I notice that /proc/kallsyms is missing in the VMs I created recently.

These VM are Ubuntu 22 guests on Ubuntu 20 or 22 hosts, created with lxd 5.0.

I could not find any useful hint after some googling. Could this is be related to how LXD configures the VM?

2

/proc/kallsyms is not about LXD configuration. This file presence is depending on the kernel compile-time option CONFIG_KALLSYMS.

You can try to check this by:

cat /boot/config-$(uname -r) | grep CONFIG_KALLSYMS

This is only about VMs, not about containers!

3

Thanks a lot for the explanation!

Indeed the 5.15.0-1028-kvm kernel that comes with the ubuntu:22.04 lxd VM images does have that flag set.

4 
root@ubuntu:/home/ubuntu# cat /boot/config-$(uname -r) | grep CONFIG_KALLSYMS
# CONFIG_KALLSYMS is not set
root@ubuntu:/home/ubuntu# uname -r
5.15.0-1020-kvm

on 5.15.0-1028-kvm it’s set?

5

on 5.15.0-1028-kvm it’s set?

I get the same result, so no it is not set:

root # cat /boot/config-$(uname -r) | grep CONFIG_KALLSYMS
# CONFIG_KALLSYMS is not set
root # uname -r
5.15.0-1028-kvm
6

so, you can try to install -generic kernel instead of -kvm in your VM.

7

In general, you are better off with linux-image-virtual inside a VM as the -kvm has a weird config set.

@amikhalitsyn, -generic is heavier as it pulls a bunch of extra modules and the CPU microcodes which are not needed in a VM.

2 Likes
8

If the -kvm flavor should grow that CONFIG_KALLSYMS, please report a bug on Launchpad (https://launchpad.net/ubuntu/+source/linux-meta-kvm/+bugs) asking for it. We’ve done that a number of times already (for other configs) :confused:

9

Thanks for the advice @sdeziel. I don’t think I’m qualified to report such a bug tough :slight_smile: I’m just trying to help debug a microk8s/dqlite issue, and my microk8s nodes happen to be on ubuntu:22.04 lxd images which have the -kvm kernel by default, and I fell into that rabbit hole.

10

I’m still trying to obtain a perf report on that vm. So I tried installing linux-virtual on the VM, updated the grub config, but it does not boot with the following error:

[    1.002064] VFS: Cannot open root device "PARTUUID=72954768-850e-45d3-a9ca-d064a700c8b5" or unknown-block(0,0): error -6
[    1.003779] Please append a correct "root=" boot option; here are the available partitions:
[    1.005180] Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(0,0)

Is there anything special to do to boot a generic kernel in a LXD virtual machine?

11

Nothing special needs to be done to boot that -virtual kernel, as long as the grub config is good, it should work. As a workaround, I’d try using root=/dev/sda2 as that is usually where the rootfs is on normal LXD VMs.

12

Still no luck. The -generic kernel does not detect any disk, apparently.
The grub config was generated by apt install linux-generic and nothing special stands out compared to the -kvm entries.

13

Could you provide the steps you’ve done to get that -kvm VM? I’d like to reproduce this locally if possible. Thanks

14

Thanks for your interest!

Here is the script I used (on a ubuntu 20.04 host with lxd 5.0 snap):

lxc init --vm ubuntu:22.04 $NAME
lxc config set $NAME limits.cpu=$CPU
lxc config set $NAME limits.memory=$RAM
lxc config device override $NAME root size=$DISK
lxc config device add $NAME eth0 nic nictype=macvlan parent=eno1 hwaddr=$ETH0MAC
lxc config device add $NAME config disk source=cloud-init:config
lxc storage volume create data $NAME-hp --type=block size=$DISK_HOSTPATH_STORAGE
lxc config device add $NAME sdb disk source=$NAME-hp pool=data
cat newvm-cloudinit.user-data | envsubst | lxc config set $NAME cloud-init.user-data -

My cloud-init script does nothing fancy: just formating sdb as ext4 and apt install of a couple of packages.

The resulting devices section of lxc config looks like this:

devices:
  config:
    source: cloud-init:config
    type: disk
  eth0:
    hwaddr: 02:00:00:xx:xx:xx
    nictype: macvlan
    parent: eno1
    type: nic
  root:
    path: /
    pool: default
    size: 50GB
    type: disk
  sdb:
    pool: data
    source: my-volume-name-hp
    type: disk

Let me know if you need more information.

15

Oh, the ubuntu: remote uses images I’m not used to. Their rootfs is at /dev/sda1 with cloudinit-rootfs LABEL. Anyway, here’s what I did to get the VM booting the -virtual/-generic kernel:

lxc shell $NAME
apt-get update && apt-get install linux-image-virtual
apt-get autopurge linux-image-kvm
rm /etc/default/grub.d/40-force-partuuid.cfg  # disable initrdless booting
update-grub
reboot
16

That did the trick, thanks a lot!

1 Like