Operation not permitted Failed to mount "proc" onto "/usr/lib/x86_64-linux-gnu/lxc/rootfs/proc"

teoli · February 18, 2024, 12:02am

Hi Community!

I am working on a small server with up-to-date Debian 12, that is hosting LXC containers and VMs. To be more specific, it is a proxmox server but I don’t think that the issue is related to px.

I’m encountering a really strange issue, that is affecting ALL unprivileged containers, new or old. Even if I create a new unpriv. container, I’m getting the same results.
The issue appeared yesterday after a mains power outage/failure.
After the server boot up, all unpriv. LXCs were (and are still) down.
I have checked the logs, and I cannot get past this error. I tried all found Information online but I can’t get it to work, I’m out of ideas that’s why I’m reaching out to you…
The error is this:

lxc-start 103 20240217230903.616 INFO     conf - ../src/lxc/conf.c:mount_autodev:1281 - Prepared "/dev"
lxc-start 103 20240217230903.616 DEBUG    conf - ../src/lxc/conf.c:lxc_mount_auto_mounts:736 - Invalid argument - Tried to ensure procfs is unmounted
lxc-start 103 20240217230903.616 TRACE    conf - ../src/lxc/conf.c:lxc_mount_auto_mounts:743 - Created procfs mountpoint under 17
lxc-start 103 20240217230903.616 DEBUG    conf - ../src/lxc/conf.c:lxc_mount_auto_mounts:759 - Invalid argument - Tried to ensure sysfs is unmounted
lxc-start 103 20240217230903.616 TRACE    conf - ../src/lxc/conf.c:lxc_mount_auto_mounts:766 - Created sysfs mountpoint under 17
**lxc-start 103 20240217230903.616 ERROR    utils - ../src/lxc/utils.c:safe_mount:1220 - Operation not permitted - Failed to mount "proc" onto "/usr/lib/x86_64-linux-gnu/lxc/rootfs/proc"**
**lxc-start 103 20240217230903.616 ERROR    conf - ../src/lxc/conf.c:lxc_mount_auto_mounts:811 - Operation not permitted - Failed to mount "proc" on "/usr/lib/x86_64-linux-gnu/lxc/rootfs/proc" with flags 14																																																															  **
**lxc-start 103 20240217230903.616 ERROR    conf - ../src/lxc/conf.c:lxc_setup:4403 - Failed to setup first automatic mounts**
**lxc-start 103 20240217230903.616 ERROR    start - ../src/lxc/start.c:do_start:1272 - Failed to setup container "103"**																																										   
lxc-start 103 20240217230903.616 TRACE    sync - ../src/lxc/sync.c:lxc_sync_wake_parent:104 - Child waking parent with sequence error
lxc-start 103 20240217230903.620 ERROR    sync - ../src/lxc/sync.c:sync_wait:34 - An error occurred in another process (expected sequence number 3)
lxc-start 103 20240217230903.620 TRACE    start - ../src/lxc/start.c:lxc_expose_namespace_environment:906 - Set environment variable LXC_USER_NS=/proc/102808/fd/17

I’ve verified the folder /usr/lib/x86_64-linux-gnu/lxc/rootfs/; It is present and it has the correct user rights. And no, proc doesn’t have to exist. I tried to create it, made no difference. On other hosts “proc” is not even present, there should be just a readme file.
I’m starting the unpriv. lxc container as root, but this is “normal behavior” in the px envrionment. Actually, the same container is starting on another debian host, with the same settings, user and rights. And as said, it occured only after the power outage.
On a healthy host, the logs look like this:

lxc-start 103 20240217231011.771 INFO     conf - ../src/lxc/conf.c:mount_autodev:1281 - Prepared "/dev"
lxc-start 103 20240217231011.772 DEBUG    conf - ../src/lxc/conf.c:lxc_mount_auto_mounts:736 - Invalid argument - Tried to ensure procfs is unmounted
lxc-start 103 20240217231011.772 TRACE    conf - ../src/lxc/conf.c:lxc_mount_auto_mounts:743 - Created procfs mountpoint under 17
lxc-start 103 20240217231011.773 DEBUG    conf - ../src/lxc/conf.c:lxc_mount_auto_mounts:759 - Invalid argument - Tried to ensure sysfs is unmounted
lxc-start 103 20240217231011.773 TRACE    conf - ../src/lxc/conf.c:lxc_mount_auto_mounts:766 - Created sysfs mountpoint under 17
**lxc-start 103 20240217231011.775 TRACE    conf - ../src/lxc/conf.c:lxc_mount_auto_mounts:820 - Mounted automount "proc" on "/usr/lib/x86_64-linux-gnu/lxc/rootfs/proc" read-write with flags 14**
lxc-start 103 20240217231011.777 TRACE    conf - ../src/lxc/conf.c:lxc_mount_auto_mounts:820 - Mounted automount "/usr/lib/x86_64-linux-gnu/lxc/rootfs/proc/sys/net" on "/usr/lib/x86_64-linux-gnu/lxc/rootfs/proc/tty" read-write with flags 4096
lxc-start 103 20240217231011.778 TRACE    conf - ../src/lxc/conf.c:lxc_mount_auto_mounts:820 - Mounted automount "/usr/lib/x86_64-linux-gnu/lxc/rootfs/proc/sys" on "/usr/lib/x86_64-linux-gnu/lxc/rootfs/proc/sys" read-write with flags 4096
lxc-start 103 20240217231011.779 TRACE    conf - ../src/lxc/conf.c:lxc_mount_auto_mounts:818 - Remounted automount "(null)" on "/usr/lib/x86_64-linux-gnu/lxc/rootfs/proc/sys" read-only with flags 4129
lxc-start 103 20240217231011.780 TRACE    conf - ../src/lxc/conf.c:lxc_mount_auto_mounts:820 - Mounted automount "/usr/lib/x86_64-linux-gnu/lxc/rootfs/proc/tty" on "/usr/lib/x86_64-linux-gnu/lxc/rootfs/proc/sys/net" read-write with flags 8192
lxc-start 103 20240217231011.781 TRACE    conf - ../src/lxc/conf.c:lxc_mount_auto_mounts:820 - Mounted automount "/usr/lib/x86_64-linux-gnu/lxc/rootfs/proc/sysrq-trigger" on "/usr/lib/x86_64-linux-gnu/lxc/rootfs/proc/sysrq-trigger" read-write with flags 4096

I have verified the virtual storage of the container, it is intact. But this should not matter, as the issue occurs with new containers too.
Smarttest passed, fsck for /dev/mapper/pve-root on / passed…
I have compared the mounts with those on the healthy server, looks almost identical, just hardware/size differences.

I suppose that maybe some file or a package got corrupted? Is this even possible, while all the rest is working as expected?!

If anyone has any ideas, I would be thankfull!

PS: I thought of rolling back to an older system backup, but I really want to know, what happened and how it can be fixed. It is also interesting to find out, what lead to this issue and how it can be avoided (as good as possible), in the feature, in case of further power loses.

Thank you!

Teo

stgraber · February 18, 2024, 5:15pm

Is the container started by an unprivileged user too?

If so, can you show grep /proc /proc/self/mountinfo on the host?
Chances are you’re hitting some kind of over-mounting protection here.

teoli · February 19, 2024, 12:04pm

Hi Stephane and thanks for your quick answer!

No. as mentioned, the container is started by root.

sure, this is the output of /proc /proc/self/mountinfo

root@c:~# grep /proc /proc/self/mountinfo
25 29 0:23 / /proc rw,relatime shared:13 - proc proc rw
37 25 0:32 / /proc/sys/fs/binfmt_misc rw,relatime shared:14 - autofs systemd-1 rw,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=3023
98 37 0:41 / /proc/sys/fs/binfmt_misc rw,nosuid,nodev,noexec,relatime shared:52 - binfmt_misc binfmt_misc rw
252 25 252:1 /tmp /proc/640 rw,relatime shared:1 - ext4 /dev/mapper/pve-root rw,errors=remount-ro

I also tried to do a manual mount, f.i. mount -t proc proc /dev/.proc

This generated no error, and I could access and list the files of proc mount in /dev/.proc/

stgraber · February 19, 2024, 6:26pm

Hmm, so that /tmp mounted on /proc/640 is very suspicious and could definitely explain some of the issues. Any idea what’s going on with that?

Does umount -l /proc/640 fix the startup issue?

teoli · February 20, 2024, 1:37pm

YES! It works.

Great hint, Thank you!

So the question is, how does this mount instruction got there?
Any idea to find out which config/program is responsible for this mount?
Because at a reboot this mount info will appear again, right?

edit: yes, I can confirm. The issue appears again after a reboot. So there is some bad config persisted somewhere…

teoli · February 20, 2024, 5:07pm

I’ve got good news! (edit: but no final solution)

I tried to find the source of the error with all possible commands and analyzes that I could think of. No success!
What I did next was to force a fsck repair during the next boot (the server is not in front of me, it is currently only remotely maintanable, so I modified the default grub entry).

Result?: No more suspicious mount like /tmp /proc/<randomNumber> after starting
LXC containers of any type are now startable!

FIY if anyone arrives here with a similar issue:
I’ve added following flags to the grub default boot entry (/etc/default/grub): fsck.mode=force fsck.repair=yes

Than I updated grub (update-grub) and rebooted.
The following reboot process took longer than expected, which is normal.
Now you should edit the grub entry back to original.

Maybe I should have done this right at beginning, don’t know why I was that hesitant
Thank you for assisting me Stephane!

edit: I take back what I said. The issue is still present after the next reboot without an fsck…

teoli · February 24, 2024, 9:39pm

Guys, I just wanted to update this topic, in case anyone lands here.
It was not a failing ssd or the data that was compromised by the power outage. It was a malware that compromised the whole OS!
Meanwhile I don’t even think, that there was a power outage, but a network lockdown due to overload by a ddos/bruteforce/etc.

It seems that there was an easy to guess root password, which was found by brute force. I know, it was my own fault regarding security in many ways, but to my defense, this password was not set by me and it was temporary, until the system will be fully set up. The OS was only about 4-5 days old, and ssh was not even exposed the whole time

This led to a malware on the system. A derivative of the Kaiji malware, I would say. It is very similar to this topic, addressed here:
Chaos Malware Quietly Evolves Persistence and Evasion Techniques – Sysdig
or here, a very detailed report of the involved files altered and the steps taken by the malware: JOE Sandbox Cloud Basic Linux Analysis Report
or here: Researchers smell a cryptomining Chaos RAT targeting Linux • The Register

The file responsible for the unwanted mount was especially /etc/profile.d/bash_cfg. I found it out by a binary search, and this was even the first hint that led to the search hits and the links posted above.

I was able to neutralize the threat by a search of all modified files in a specific timespan, and also by the help of the report in the second link. After that everything worked again as expected.

However, I wanted to be sure that no other undetected/modified files were left behind, and I reinstalled the OS from scratch.

FIY: the forced fsck, mentioned in the post before, did not even run at startup because the initramfs from Proxmox didn’t even have a fsck included(?) at least there was a startup log entry that suggested this (or maybe was this also because of the malware)?

Anyhow, it was a painful experience, but I learned a lot in many ways and I am happy I could find the root cause instead of just reinstalling the OS.

Reminder for everyone: Don’t play around with security/fire

BR and thank you for the containers!