Container performance issue

Hi, today I found my containers inside LXD server becomes very slow, it means that any command even login becomes very slow, the system takes a long time to repsonse, not only one container, but all of them.

One of intuitive description:

htop from host:

I’ve check the system load, iowait, is all ok, backed storage is ZFS, running 200+ container, LXD 4.17
But I found that VirtualMachine is fine, every command repsonse quickly, only container.

Any idea will be welcome, thank you.

What’s LXD doing? Are you getting a lot of output if you run lxc monitor?

Thank you for your help.
I have a script to read the ct’s state from pylxd every 3 minute, I thought the problem came from this script, then I stop the script and wait for about 30minute, the top still showing that LXD’s process is using a lot of CPU, and there is nothing in lxc monitor, so I dont know what’s happening, any idea?

lxc monitor:

htop:

I’ve checked /var/snap/lxd/common/lxd/logs/lxd.log too. and it seems nothing needs to worried about.

What version of LXD is that?

4.17
image

Ok, that’s pretty odd. Can you do:

This will get us a basic trace of every goroutine running in LXD.

Emm, I can’t read it :rofl:

Can you show lxc operation list?


It looks some operations hangs long time ago for some reason, and I don’t know where it is or what they are doing.
I asked about those operation before, but I have 200+ containers running so I couldn’t find them…
:frowning: