This WE, I played with LXC/LXD containers and ElasticSearch inside my homelab for sake of learning.
So I use as usual Fedora as “OS” for the container, I use limits for memory (6GB) and CPU (4vCPU) and I setup a single node of Elasticsearch, but once I want to start the service, it fails miserably. I looked to find some clue on why it failed and I found the process seems to be OOM-killed, it looks like the JVM memory heap allocation was completely borked. I checked the process closely and it allocate “shit-load” of memory. I guess it tried to size the heap according to my host memory (64GB of RAM).
I didn’t understand, so I tried a virtual machine with the same hardware limits and it works perfectly fine, the JVM has well adjusted its heap, so WTF…
I continue to search in order to find an explaination but no luck until I change a setting, the famous
security.nesting that I use because of the recent changes in systemd concerning sandboxing settings applied to
systemd-networkd unit (and others too I guess). Once I changed this setting to
false and reboot my container, the Elasticseach process start without any problems, the heap allocation looks fine
I checked the
/proc/meminfo file to check LXCFS “virtualization” of memory values and it looks fine, the limits were correct.
Is it something intended to have process which can “override” their view of
/proc when using
security.nesting ? I found issues on Github about this kind of behavior with Docker with this setting applied, but why when this parameter is not set, the application seems to behave correctly ? I could read that JVM use the meminfo file, so it is very confusing
Any ideas ?