I have a Rancher based Kubernetes cluster that consists of nodes that are LXD containers. I have been able to deploy services to these nodes successfully. However, Rancher seems to think that each of these nodes has all the resources of the host the LXD container is running on.
For example, I launched a few worker node containers. These were ensured to have limited CPU cores using lxc config set my-container limits.cpu 4 . This was verified by running grep -c processor /proc/cpuinfo in the container, which returned 4 as expected. However, Rancher shows the same node as having 56 cores, which is the number of cores on the bare metal where the LXD containers are running. This is shown in the screenshot below:
All the nodes shown in the screenshot were CPU limited as mentioned above, but they are listed as having 56 cores by Rancher. This is problematic, given that many of these containers are on the same host.
I am running LXD 3.0.4 (from the 3.0/stable channel). Based on what I have read about LXCFS, I believe it may solve my problem. However, I am not sure if LXCFS is already running on my snap installed LXD. I see the following process on my machine when I run ps aux | grep lxcfs:
/var/snap/lxd/common/var/lib/lxcfs shown above points to /var/snap/lxd/common/shmounts/lxcfs, which does not exist (/var/snap/lxd/common/shmounts/ is an empty directory).
As per this, LXCFS will ensure that /sys/devices/system/cpu/online is container aware. It seems to me that Rancher obtains resource information from this location, which is why I believe a working install of LXCFS will solve my problem. Please let me know if you have any thoughts on this.
How do I use LXCFS with my snap installed LXD? The instructions here do not seem to be specific to a snap installation.
Anything else I should be considering? Appreciate all feedback and suggestions.
inspecting files under proc by running nsenter -t $(pgrep daemon.start) -m -- cat /var/snap/lxd/common/shmounts/lxcfs/proc/cpuinfo I see all 56 cores of my server listed.
Is this correct behavior?
How do I verify if LXCFS is correctly mounting a container-aware version of these dirs into containers?
Certainly it’s correct since this path don’t refer the container at all, and the limit you set is on a specific container.
If in a container you don’t see a reduced CPU number in /proc/cpuinfo, it’s a problem.
I can’t see it on snap LXD 3.16. I have an old LXD version installed through APT (so 3.03 with lxcfs 3.03) and /proc/cpuinfo in a container is showing a reduced proc count.
So it don’t seem to come from an lxcfs recent enhancement;
So it seems that your software don’t use /proc/cpuinfo to count processors.
A quick search in the rancher issues leads to this.
That is an issue I posted on Rancher’s Github to try and figure this out before I found out about LXCFS. The response to my post is not a real solution to the problem (the responder didn’t seem to fully understand my question).
So, looking at /proc/cpuinfo in a running container shows:
root@server:~# lxc exec test -- grep -c "processor" /proc/cpuinfo
1
Which is what I expected. However:
root@server:~# lxc exec test -- cat /sys/devices/system/cpu/online
0-55
That is not something I expected, since the container should only have access to one core.
This says that LXCFS will mount /sys/devices/system/cpu/online in a container aware manner, but that doesn’t seem to be the case here. Any way I can force this to happen?
After looking a bit at that, I should add that it’s pretty normal in fact that /sys/devices/system/cpu/online is not container-aware on LXD 3.03 (Apt version) since the feature was not developped. The problem occurs also on LXD 3.16 and here it seems less normal.
Nonetheless, it’s not certain that fixing this problem would fix the Rancher issue.
There are other ways of looking at Cpu info, such as this that are just not covered by lxcfs.
