Disable logging

LXD
1

I noticed that on my setup (12 VMs) lxd --logfile takes one of the top cpu usage in total

lxd --logfile /var/snap/lxd/common/lxd/logs/lxd.log --group lxd

How can I disable this logging?

2

The process is the LXD daemon itself.

Are you seeing a lot of content in the /var/snap/lxd/common/lxd/logs/lxd.log itself?

You can peek inside to see what the LXD process is doing by:


    lxc config set core.debug_address 127.0.0.1:8444
    curl http://127.0.0.1:8444/debug/pprof/goroutine?debug=2
3

It’s under 1mb, but look at htop when a VM running:

image
image919Ă—103 20.8 KB

4

Anything useful in the log file?

Given it gets reset on lxd reload, 1mb is quite a lot by the sounds of it.

5

Full of this warning:

time="2023-06-09T11:19:39Z" level=warning msg=
"Unable to use virtio-fs for config drive, using 9p as a fallback" 
err="Stateful migration unsupported"
instance=worker-1 
instanceType=virtual-machine project=default

Here is content of zfs list -t all

NAME                                                                                             USED  AVAIL     REFER  MOUNTPOINT
zfspool                                                                                         24.9G   264G       24K  legacy
zfspool/buckets                                                                                   24K   264G       24K  legacy
zfspool/containers                                                                                24K   264G       24K  legacy
zfspool/custom                                                                                    24K   264G       24K  legacy
zfspool/deleted                                                                                  144K   264G       24K  legacy
zfspool/deleted/buckets                                                                           24K   264G       24K  legacy
zfspool/deleted/containers                                                                        24K   264G       24K  legacy
zfspool/deleted/custom                                                                            24K   264G       24K  legacy
zfspool/deleted/images                                                                            24K   264G       24K  legacy
zfspool/deleted/virtual-machines                                                                  24K   264G       24K  legacy
zfspool/images                                                                                  2.98G   264G       24K  legacy
zfspool/images/f2a2765917eb0d5d8881e2b3389636671837aee285cf39c8f82ba0e98cb08b95                   27K   100M       26K  legacy
zfspool/images/f2a2765917eb0d5d8881e2b3389636671837aee285cf39c8f82ba0e98cb08b95@readonly           1K      -       26K  -
zfspool/images/f2a2765917eb0d5d8881e2b3389636671837aee285cf39c8f82ba0e98cb08b95.block           2.98G   264G     2.98G  -
zfspool/images/f2a2765917eb0d5d8881e2b3389636671837aee285cf39c8f82ba0e98cb08b95.block@readonly     0B      -     2.98G  -
zfspool/virtual-machines                                                                        21.9G   264G       24K  legacy
zfspool/virtual-machines/worker-1                                                               1.13G  1.87G     7.46M  legacy
zfspool/virtual-machines/worker-1@snapshot-snap0                                                1.12G      -     1.13G  -
zfspool/virtual-machines/worker-1.block                                                          704M   264G     3.65G  -
zfspool/virtual-machines/worker-1.block@snapshot-snap0                                          7.96M      -     3.62G  -

Update: I’m running 10-15 LXD VMs in parallel, each have their own Qemu running inside
Each Qemu use different vsock but not the host LXD VM since they is no support to specify vsock

lxc config get <instance> volatile.vsock_id doesn’t work according to @stgraber and should NOT be used from clients

6

Warnings are fine.

I suggest stopping the VMs, see if the CPU usage goes down.

Then you know its something the VMs are doing to cause it.

Also suggest taking some snapshots of the output of the goroutine debug output mentioned above with no VMs running and LXD just freshly reloaded, compared to one VM running and one more etc until you start seeing the issue and then we can correlate the running go routines with the issue.

Suggest also stop running the nested VMs until and try and isolate the cause.

7

Is the lxd load on the host or the guest?

8

on the host

9

With CPUs stopped, the usage also goes down

11

why we may getting this warning/err:

Unable to use virtio-fs for config drive, using 9p as a fallback

Due to high i/o, I think we would benefit from virtio-fs

12

You’re missing virtiofsd:

1 Like
13

Am I supposed install it separately from LXD?
No doc mentions anything similar

Any usage guide would be appreciated

14

Its included in the snap package, what are you using?

15

I’m using snap package:

ubuntu@ubuntu:~$ file /usr/lib/qemu/virtiofsd
/usr/lib/qemu/virtiofsd: ELF 64-bit LSB pie executable, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, stripped

It’s here.

16 
which virtiofsd

Returns empty stings, that’s may be the reason.
Maybe thats how Go’s exec.LooPath work?

17

https://cs.opensource.google/go/go/+/master:src/os/exec/lp_unix.go;l=44?q=LookPath&ss=go%2Fgo:src%2Fos%2Fexec%2F

It searches within $PATH, but /usr/lib/qemu/ isn’t under $PATH

18

Yes, snap doesn’t use anything from the host OS.

Ah, its not because of that its because you’ve got migration.stateful enabled.
But virtiofs doesn’t support stateful migration.

1 Like
19

Is there a reason why virtiofs doesn’t support stateful and p9 does?

20

It’s a QEMU side limitation. I don’t know the exact reason for it but I certainly hope that limitation goes away in the near future as virtiofs is so much better…

1 Like