I noticed that on my setup (12 VMs) lxd --logfile
takes one of the top cpu usage in total
lxd --logfile /var/snap/lxd/common/lxd/logs/lxd.log --group lxd
How can I disable this logging?
I noticed that on my setup (12 VMs) lxd --logfile
takes one of the top cpu usage in total
lxd --logfile /var/snap/lxd/common/lxd/logs/lxd.log --group lxd
How can I disable this logging?
The process is the LXD daemon itself.
Are you seeing a lot of content in the /var/snap/lxd/common/lxd/logs/lxd.log
itself?
You can peek inside to see what the LXD process is doing by:
lxc config set core.debug_address 127.0.0.1:8444
curl http://127.0.0.1:8444/debug/pprof/goroutine?debug=2
Anything useful in the log file?
Given it gets reset on lxd reload, 1mb is quite a lot by the sounds of it.
Full of this warning:
time="2023-06-09T11:19:39Z" level=warning msg=
"Unable to use virtio-fs for config drive, using 9p as a fallback"
err="Stateful migration unsupported"
instance=worker-1
instanceType=virtual-machine project=default
Here is content of zfs list -t all
NAME USED AVAIL REFER MOUNTPOINT
zfspool 24.9G 264G 24K legacy
zfspool/buckets 24K 264G 24K legacy
zfspool/containers 24K 264G 24K legacy
zfspool/custom 24K 264G 24K legacy
zfspool/deleted 144K 264G 24K legacy
zfspool/deleted/buckets 24K 264G 24K legacy
zfspool/deleted/containers 24K 264G 24K legacy
zfspool/deleted/custom 24K 264G 24K legacy
zfspool/deleted/images 24K 264G 24K legacy
zfspool/deleted/virtual-machines 24K 264G 24K legacy
zfspool/images 2.98G 264G 24K legacy
zfspool/images/f2a2765917eb0d5d8881e2b3389636671837aee285cf39c8f82ba0e98cb08b95 27K 100M 26K legacy
zfspool/images/f2a2765917eb0d5d8881e2b3389636671837aee285cf39c8f82ba0e98cb08b95@readonly 1K - 26K -
zfspool/images/f2a2765917eb0d5d8881e2b3389636671837aee285cf39c8f82ba0e98cb08b95.block 2.98G 264G 2.98G -
zfspool/images/f2a2765917eb0d5d8881e2b3389636671837aee285cf39c8f82ba0e98cb08b95.block@readonly 0B - 2.98G -
zfspool/virtual-machines 21.9G 264G 24K legacy
zfspool/virtual-machines/worker-1 1.13G 1.87G 7.46M legacy
zfspool/virtual-machines/worker-1@snapshot-snap0 1.12G - 1.13G -
zfspool/virtual-machines/worker-1.block 704M 264G 3.65G -
zfspool/virtual-machines/worker-1.block@snapshot-snap0 7.96M - 3.62G -
Update: I’m running 10-15 LXD VMs in parallel, each have their own Qemu running inside
Each Qemu use different vsock but not the host LXD VM since they is no support to specify vsock
lxc config get <instance> volatile.vsock_id
doesn’t work according to @stgraber and should NOT be used from clients
Warnings are fine.
I suggest stopping the VMs, see if the CPU usage goes down.
Then you know its something the VMs are doing to cause it.
Also suggest taking some snapshots of the output of the goroutine debug output mentioned above with no VMs running and LXD just freshly reloaded, compared to one VM running and one more etc until you start seeing the issue and then we can correlate the running go routines with the issue.
Suggest also stop running the nested VMs until and try and isolate the cause.
Is the lxd load on the host or the guest?
on the host
With CPUs stopped, the usage also goes down
why we may getting this warning/err:
Unable to use virtio-fs for config drive, using 9p as a fallback
Due to high i/o, I think we would benefit from virtio-fs
You’re missing virtiofsd:
Am I supposed install it separately from LXD?
No doc mentions anything similar
Any usage guide would be appreciated
Its included in the snap package, what are you using?
I’m using snap package:
ubuntu@ubuntu:~$ file /usr/lib/qemu/virtiofsd
/usr/lib/qemu/virtiofsd: ELF 64-bit LSB pie executable, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, stripped
It’s here.
which virtiofsd
Returns empty stings, that’s may be the reason.
Maybe thats how Go’s exec.LooPath work?
It searches within $PATH, but /usr/lib/qemu/
isn’t under $PATH
Yes, snap doesn’t use anything from the host OS.
Ah, its not because of that its because you’ve got migration.stateful
enabled.
But virtiofs doesn’t support stateful migration.
Is there a reason why virtiofs doesn’t support stateful
and p9
does?
It’s a QEMU side limitation. I don’t know the exact reason for it but I certainly hope that limitation goes away in the near future as virtiofs is so much better…