Thanks!
This command worked for me:
lxc launch ubuntu:18.04 wireguard -c linux.kernel_modules=wireguard
Any idea what parts need to be installed inside the container? I installed everything, but it seem like a smaller number of packages would be needed, since the host already have the kernal packages installed.