How to allocate cores which are in isolcpus list of host

I have a list of isolated cpus in kernel boot parameter as like isolcpus=“4-15”.
I’d like to pin some of isolated cpus to lxd container as like limits.cpu: 4-7
Inside the container, I’d like to use those isolated and pinned cpus for a specific purpose.
According to my experiment, my dpdk app (i.e. dpdk sample app, l3fwd) fails to run.
Is there any way to pin isolated cpus to lxd container?

I appreciate any comments.

Maybe @brauner remembers the rules around isolcpus, but I remember them behaving very weirdly when combined with cgroups. limits.cpu: 4-7 should cause LXD to tell liblxc to configure the cpuset cgroup to allow those cores in the container, but whether the kernel actually complies is another matter.

@stgraber, thank you for your comment.
According to my trials, I’ve not found any effective way to allocate isolated CPUs to lxd container as shown below. If any, please let me know…

ubuntu@r9:~$ lxc launch ubuntu:bionic bionic
Creating bionic
Starting bionic
ubuntu@r9:~$ lxc config set bionic limits.cpu 3,7,11,15,2,6,10,14
ubuntu@r9:~$ cat /proc/cmdline
BOOT_IMAGE=/boot/vmlinuz-4.15.0-51-generic root=UUID=341b9db3-0131-4ac8-92ff-1f9acc544ca4 ro hugepagesz=2M hugepages=32768 isolcpus=1,5,9,13,17,21,25,29,2,6,10,14,18,22,26,30 nohz_full=1,5,9,13,17,21,25,29,2,6,10,14,18,22,26,30 rcu_nocbs=1,5,9,13,17,21,25,29,2,6,10,14,18,22,26,30 iommu=pt intel_iommu=on
ubuntu@r9:~$ numactl -H
available: 4 nodes (0-3)
node 0 cpus: 0 4 8 12 16 20 24 28
node 0 size: 48278 MB
node 0 free: 30158 MB
node 1 cpus: 1 5 9 13 17 21 25 29
node 1 size: 48382 MB
node 1 free: 30697 MB
node 2 cpus: 2 6 10 14 18 22 26 30
node 2 size: 48382 MB
node 2 free: 30561 MB
node 3 cpus: 3 7 11 15 19 23 27 31
node 3 size: 48361 MB
node 3 free: 29784 MB
node distances:
node 0 1 2 3
0: 10 21 21 21
1: 21 10 21 21
2: 21 21 10 21
3: 21 21 21 10
ubuntu@r9:~$ lxc exec bionic – cat /proc/cpuinfo | grep processor
processor : 0
processor : 1
processor : 2
processor : 3
ubuntu@r9:~$ echo “3,7,11,15,2,6,10,14” | sudo tee /sys/fs/cgroup/cpuset/lxc/bionic/cpuset.cpus
3,7,11,15,2,6,10,14
tee: /sys/fs/cgroup/cpuset/lxc/bionic/cpuset.cpus: Permission denied
ubuntu@r9:~$ echo “3,7,11,15,2,6,10,14” | sudo tee /sys/fs/cgroup/cpuset/lxc/bionic/cpuset.effective_cpus
3,7,11,15,2,6,10,14
tee: /sys/fs/cgroup/cpuset/lxc/bionic/cpuset.effective_cpus: Invalid argument
ubuntu@r9:~$

well, I tried to add isolated cpus to lxc’s cpuset first and then add those cpus to a specific container. It seems to work!

root@r9:/sys/fs/cgroup/cpuset/lxc# cat cpuset.cpus
0,3-4,7-8,11-12,15-16,19-20,23-24,27-28,31
root@r9:/sys/fs/cgroup/cpuset/lxc# echo 0-31 | tee cpuset.cpus > /dev/null
root@r9:/sys/fs/cgroup/cpuset/lxc# echo 0-31 | tee cpuset.effective_cpus > /dev/null
tee: cpuset.effective_cpus: Invalid argument
root@r9:/sys/fs/cgroup/cpuset/lxc# cat cpuset.cpus
0-31
root@r9:/sys/fs/cgroup/cpuset/lxc# cat cpuset.effective_cpus
0-31
root@r9:/sys/fs/cgroup/cpuset/lxc# cd bionic
root@r9:/sys/fs/cgroup/cpuset/lxc/bionic# cat cpuset.cpus
3,7,11,15
root@r9:/sys/fs/cgroup/cpuset/lxc/bionic# echo “3,7,11,15,2,6,10,14,18,22,26,30” | sudo tee cpuset.cpus > /dev/null
root@r9:/sys/fs/cgroup/cpuset/lxc/bionic# cat cpuset.cpus
2-3,6-7,10-11,14-15,18,22,26,30
root@r9:/sys/fs/cgroup/cpuset/lxc/bionic# lxc exec bionic – cat /proc/cpuinfo |grep processor
processor : 0
processor : 1
processor : 2
processor : 3
processor : 4
processor : 5
processor : 6
processor : 7
processor : 8
processor : 9
processor : 10
processor : 11
root@r9:/sys/fs/cgroup/cpuset/lxc/bionic# echo “3,7,11,15,2,6,10,14,18,22,26,30” | sudo tee cpuset.effective_cpus > /dev/null
tee: cpuset.effective_cpus: Invalid argument
root@r9:/sys/fs/cgroup/cpuset/lxc/bionic# cat cpuset.effective_cpus
2-3,6-7,10-11,14-15,18,22,26,30
root@r9:/sys/fs/cgroup/cpuset/lxc/bionic#

I think you need to manually assign.
One should also note that isolcpus is deprecated.
At least the boot parameter.

@brauner, ok. thank you for your comment.
I’ll take a look into cpuset manual page to follow up the changed kernel parameters.

for others’ information, refer to below URL.
https://www.kernel.org/doc/html/v4.18/admin-guide/kernel-parameters.html

isolcpus= [KNL,SMP,ISOL] Isolate a given set of CPUs from disturbance.
[Deprecated - use cpusets instead]
Format: [flag-list,]