Launching VMs without CPU limit

LXD
1

I’m trying to start VM instances without CPU limit. I was looking at how the qemu config gets generated in lxd/instance/drivers/driver_qemu.go and based on that I was using this:

lxc config set $instancename limits.cpu 0-$(($(nproc)-1))

(Yes, this is not very good, because nproc needs to be executed on the same host where the instance is running and limits.cpu needs to be set again after moving the instance to a machine with different number of CPUs. But this was the best solution I could come up with.)

The problem is today it stopped working:

root@oth-node1:~# lxc start oth-ipsec
Error: Failed to run: forklimits limit=memlock:unlimited:unlimited fd=3 -- /snap/lxd/23243/bin/qemu-system-x86_64 -S -name oth-ipsec -uuid 39ea0d17-f911-4801-872e-5608dd0c697e -daemonize -cpu host,hv_passthrough,topoext -nographic -serial chardev:console -nodefaults -no-user-config -sandbox on,obsolete=deny,elevateprivileges=allow,spawn=allow,resourcecontrol=deny -readconfig /var/snap/lxd/common/lxd/logs/oth-ipsec/qemu.conf -spice unix=on,disable-ticketing=on,addr=/var/snap/lxd/common/lxd/logs/oth-ipsec/qemu.spice -pidfile /var/snap/lxd/common/lxd/logs/oth-ipsec/qemu.pid -D /var/snap/lxd/common/lxd/logs/oth-ipsec/qemu.log -smbios type=2,manufacturer=Canonical Ltd.,product=LXD -runas lxd: : Process exited with non-zero value 1
Try `lxc info --show-log oth-ipsec` for more info
root@oth-node1:~# lxc info --show-log oth-ipsec
Name: oth-ipsec
Status: STOPPED
Type: virtual-machine
Architecture: x86_64
Location: oth-node1
Created: 2022/05/04 16:51 +07
Last Used: 2022/06/24 07:02 +07

Log:

qemu-system-x86_64:/var/snap/lxd/common/lxd/logs/oth-ipsec/qemu.conf:224: CPU is already assigned to node-id: 0

If I remove the limits.cpu setting then the instance starts:

root@oth-node1:~# before="$(lxc config show oth-ipsec )"
root@oth-node1:~# lxc config unset oth-ipsec limits.cpu
root@oth-node1:~# diff -U1 <(echo "$before") <(lxc config show oth-ipsec)
--- /dev/fd/63	2022-07-06 12:04:27.076003913 +0700
+++ /dev/fd/62	2022-07-06 12:04:27.076003913 +0700
@@ -9,3 +9,2 @@
   image.variant: salt
-  limits.cpu: 0-47
   limits.memory: 2GB
root@oth-node1:~# lxc start oth-ipsec

But that takes me back to the default 1CPU limit:

$ ssh oth-ipsec nproc
1

What is the best way to start qemu VM instances without limiting the number of CPUs?

(When starting LXC container instances, the default is to not limit the CPUs, that’s great, this problem is only with VM instances.)

2

I managed to find a qemu.conf which was generated earlier and most likely still working. And a newly generated not working config for comparison. From the [numa] sections the socket-id is now missing and the numbering of node-id is different.

I can’t attach text files and copypasting into the message would be too big, so the qemu configs can be found here:

3

This line looks sus: https://github.com/lxc/lxd/blob/lxd-5.3/lxd/instance/drivers/driver_qemu_templates.go#L466

{key: "node-id", value: fmt.Sprintf("%d", numa.socket)},

Shouldn’t that be changed to this:?

{key: "node-id", value: fmt.Sprintf("%d", numa.node)},
{key: "socket-id", value: fmt.Sprintf("%d", numa.socket)},
4

For your particular scenario I believe just specifying the total amount of CPUs will achieve what you want, e.g.

lxc config set $instancename limits.cpu $(($(nproc)-1))

I don’t believe there is a way to specify that the limit match the host CPU count currently. You could open an issue at Issues · lxc/lxd · GitHub to discuss adding such a mode.

5

Perhaps this is a regression introduced with the template rework @tarruda did in LXD 5.3.

@stgraber does this seem like a bug to you? Thanks

6

Yes, used lxc config set ${instancename:?} limits.cpu=$(nproc) on the affected instances and that made it work.

What was there before:

limits.cpu: 0-47

What’s there now:

limits.cpu: 48

When using 48, then in addCPUMemoryConfig() that just sets sockets and threads to 1, which does not match the host’s NUMA topology.

Using 0-47 goes to the else branch and LXD passes on the host’s NUMA topology to the VM instance, which is a great feature and exactly what I wanted.

7

@PaulT Your’e right, I missed the “socket-id” key when converting the template. Currently adding a test case for this scenario and will submit a fix soon.

2 Likes
8

@PaulT I’ve pushed a fix to https://github.com/tarruda/lxd/tree/fix-wrong-numa-template.

Can you test this branch with your complete LXD configuration?

9

Feel free to put up a PR for us to review too, thanks.

10

I’ve compared with the old template code and indeed confirmed I made a mistake when creating the config object.

1 Like
11

This looks good. The [numa] sections in qemu.conf look different than with LXD 5.2, but the new one generated from the fix-wrong-numa-template branch is correct. Here is the generated qemu.conf from that branch:

With LXD 5.2 the node-ids went up to 3, but that’s a machine with 2 physical CPUs, so node-id should be only 0 and 1. The memdev settings look better too, in LXD 5.2 it went up to mem3, but the fix-wrong-numa-template branch generates until mem1 only, which is correct.

For context, this machine has 2 physical Xeon E5-2670 v3 CPUs, each CPU has 12 cores and each core has 2 threads.

2 Likes