For additional context on why such issues happen and are so hard to recover from.
LXCFS is a FUSE filesystem that’s exposed in all containers. This comes with some issues. We can’t detach/re-attach to a FUSE filesystem, so killing/restarting the lxcfs binary will break the FUSE mount and anything using it. Additionally, there are no good mechanisms to push/pull mounts from containers, so LXCFS dying doesn’t remove any of those mounts and re-starting it can’t inject them back either.
Because we need to be able to do bugfix, and more critically, security fixes to LXCFS on a running system without breaking all the containers. We have a clever design in place where 99% of what LXCFS does is stuffed into an internal library (liblxcfs.so). The lxcfs binary itself is just a loader for that library as well as a signal handler so when receiving SIGUSR1 it will unload the current copy of the library, load the new copy and keep doing its job.
This means we need to be extremely careful about never breaking backward compatibility in the library as we may be running from a binary that’s several years older than the library is (think upgrading from LXCFS 2.0 to 4.0).
In this particular case, the issue was that the filesystem on which lxcfs was started has since long been unmounted, when the new library is loaded, it would attempt to chdir to that path, fail and crash. This is why only a limited number of users will ever hit this case. The fix we’re rolling out effectively ignores that particular case as that error shouldn’t be fatal.
Additionally a follow-up set of fixes will be pushed to further harden the library reloading code so that even if everything goes terribly wrong, lxcfs will be left running, albeit in a mode where it only shows the host values (similar to having it unmounted from all containers).