What is the best way to use numactl or taskset and chrt in lxd which cpus are isolated from the host

LXD
1

I’m not able to pins my isolated cpu cores in my lxd container, which there are isolated on my host, from my grub configuration.

# cat /proc/cmdline 
BOOT_IMAGE=/vmlinuz-5.3.0-46-generic root=/dev/mapper/os_local_vg-root ro BOOTIF=01-38-68-dd-14-d3-s2 console=tty1 console=ttyS1,115200n8 nomodeset net.ifnames=0 biosdevname=0 processor.max_cstate=1 intel_idle.max_cstate=0 intel_pstate=disable isolcpus=4-23 nohz=on nohz_full=4-23 rcu_nocbs=4-23 skew_tick=1 idle=poll nosoftlockup

I tested some different configurations, none works correctly.

I’d like to use the isolated cpu cores with isolcpus grub parameter, althought is deprecated it seems to be the only way (I found) to completely isolate cpus on ubuntu 18.04.

I tested with cset shield… but there is a bug or in conflict with other cpusets. I open a issue on the project

# cset shield --cpu 10-15 --kthread on test

cset: --> failed to create shield, hint: do other cpusets exist?
cset: **> [Errno 22] Invalid argument

In fact I cannot assign isolate cpus cores with ‘lxc config set test limits.cpu 10-10’ from which cpus cores are isolated.

The only working way I found, it’s to override the cpuset container configuration like

echo 0-23 | tee /sys/fs/cgroup/cpuset/...test/cpuset.cpus

With that I can see whole cpus inside my lxd, and the isolated one seems not to be used by my container itself, and I can use numactl or tasket to set the affinity of my process.

But the problem with that is, if there are any modification on a container in my host, like just launch another one or delete one, the configuration of the cgroup is overridden by the default configuration of lxd…
Is there a way to permanently modifying that to ensure that my cpuset configuration isn’t altered ?

Or can I create a new cpuset like that, and assign it in addition of the existing one on my container, to be able to use with numactl ?

mkdir /dev/cpuset
mount -t cpuset none /dev/cpuset/
cd /dev/cpuset/
mkdir fx-cpus
cd fx-cpus/
echo 5-23 > cpuset.cpus
echo 0-1 > cpuset.mems

Thanks

2

any advices ?

3

Sorry, isolcpus combined with the normal cpusets give me a headache every time :slight_smile:

You can at least pin your container using limits.cpu so it doesn’t get balanced or modified but that’s unlikely to help you much with isolcpus as those are excluded from our cpuset.

4

Thanks for your feedback. So I’ve probably no choice, doing it in an old school bare metal way :wink:

5

Maybe just one last thing.
Is there a way to overwrite directly inside the db, the number of cores for a specific container, to fix manually the cpuset configuration ?
Or another way to avoid the rewrite of the cgroup ?
Thanks.

6

limits.cpu can be set to an exact pin, but that won’t do you much good since isolated CPUs cannot be used with cpuset at the kernel level.

7

(That is, LXD’s own cgroup won’t have those CPUs so we can’t then pass it to children)