Normally when upgrading LXD the VMs don’t need to be restarted.
However there was a bug fix between LXD 4.x and LXD 5.0.1 where the VM agent protocol changed from being incorrectly double TLS encrypted to correctly just being encrypted once with TLS.