Lxd daemon high cpu usage processing inotify watch on /dev/net/tun

tomp · November 24, 2022, 10:38am

Yes indeed. It just seems odd to me that watching a directory will see WRITES to files within it.
Anyway its a side topic and not really relevant, as its clearly a problem.

tomp · November 24, 2022, 1:59pm

Got a fix here:

github.com/lxc/lxd

fsmonitor: Don't watch for WRITE events (do not merge)

lxc:master ← tomponline:tp-inotify

opened 01:40PM - 24 Nov 22 UTC

tomponline

+85 -53

Switch away from `github.com/fsnotify/fsnotify` package to `k8s.io/utils/inotify…`. There is an unfortunate behaviour in the fsnotify package in that it watches for all events on a path, and doesn't give the user the option of only watching for specific events. This means that when watching, for example on /dev/net, any writes to /dev/net/tun will cause a WRITE event to be sent to LXD. As you can image, for high traffic interfaces this can cause many events and causes LXD to use a lot of CPU. This is equally the case for other devices that can be written to, such as block devices. The issue is identified in the upstream package (https://github.com/fsnotify/fsnotify/issues/7) but there doesn't seem to be any consensus on how to move forward so the issue has languished. LXD however doesn't need cross-platform support that fsnotify package offers and so can use a lightweight wrapper around the inotify feature (which is what we use with fsnotify anyway). There used to be an inotify wrapper at golang.org/x/exp/inotify which has subsequently moved to gopkg.in/fsnotify.v0. Unfortunately that doesn't support go mod and so isn't usable by LXD. Its also not been updated since 2015. There is however a fork of golang.org/x/exp/inotify at https://github.com/kubernetes/utils/tree/master/inotify which has seen some minor maintenance this year. This package also has go mod support. This commit also makes the following changes: - Avoids the need to make a stat syscall to detect if event is for a directory by using the InIsdir flag in the event Mask. - Avoids calling every registered handler for every new directory and instead performs a path prefix match. - Improves comments and readability. - Cleans the path coming from the event to ensure comparison with registered handlers works. - Reduces repitition in event action logic. Fixes https://github.com/lxc/lxd/issues/11151

Aleks · February 24, 2023, 11:12am

has this been merged? released? if yes, in which version? I have just upgraded to latest 5.10/stable and I’m seeing no improvement in CPU usage.

tomp · February 24, 2023, 1:38pm

It was in LXD 5.9 and LXD 5.0.2

tomp · February 24, 2023, 1:39pm

Looks like its something else causing the issue then.

Can you re-run your initial analysis and confirm?

Aleks · February 24, 2023, 2:05pm

yes, you are right, stracing lxd shows almost no activity, it must be something else then.