Operation not permitted Failed to mount "proc" onto "/usr/lib/x86_64-linux-gnu/lxc/rootfs/proc"

Guys, I just wanted to update this topic, in case anyone lands here.
It was not a failing ssd or the data that was compromised by the power outage. It was a malware that compromised the whole OS! :dizzy_face:
Meanwhile I don’t even think, that there was a power outage, but a network lockdown due to overload by a ddos/bruteforce/etc.

It seems that there was an easy to guess root password, which was found by brute force. I know, it was my own fault regarding security in many ways, but to my defense, this password was not set by me and it was temporary, until the system will be fully set up. The OS was only about 4-5 days old, and ssh was not even exposed the whole time :man_facepalming:

This led to a malware on the system. A derivative of the Kaiji malware, I would say. It is very similar to this topic, addressed here:
Chaos Malware Quietly Evolves Persistence and Evasion Techniques – Sysdig
or here, a very detailed report of the involved files altered and the steps taken by the malware: JOE Sandbox Cloud Basic Linux Analysis Report
or here: Researchers smell a cryptomining Chaos RAT targeting Linux • The Register

The file responsible for the unwanted mount was especially /etc/profile.d/bash_cfg. I found it out by a binary search, and this was even the first hint that led to the search hits and the links posted above.

I was able to neutralize the threat by a search of all modified files in a specific timespan, and also by the help of the report in the second link. After that everything worked again as expected.

However, I wanted to be sure that no other undetected/modified files were left behind, and I reinstalled the OS from scratch.

FIY: the forced fsck, mentioned in the post before, did not even run at startup because the initramfs from Proxmox didn’t even have a fsck included(?) at least there was a startup log entry that suggested this (or maybe was this also because of the malware)?

Anyhow, it was a painful experience, but I learned a lot in many ways :smiley: and I am happy I could find the root cause instead of just reinstalling the OS.

Reminder for everyone: Don’t play around with security/fire :sweat_smile:

BR and thank you for the containers!

1 Like