Slowness of rarely used processes inside of containers

I’m observing a very strange problem that is related to the performance degradation of some processes inside the containers. For example, I have Apache that is running inside each container along with some other processes. In contrast to the other processes, Apache is a really rarely used process. The customers connect to it only when it is needed to change the configuration of the “main” application. The problem is that after a month or more when Apache was not used, it takes a lot of time to connect to it and perform some operations.
The same slowness happens when I try to connect to the container (lxc exec bash).
The interesting moment is that the second attempt of “lxc exec …” is performed fast.
The host and containers have enough CPU and RAM resources. The swap is configured but not used.
I didn’t see any container’s CPU throttling counter jumps or IO activity during the ‘slowness’.
I am using LXD 3.0.3, Ubuntu 18.04

Any idea how to fix this problem?